ceph-csi
ceph-csi copied to clipboard
ceph
failed to use a balanced ip with haproxy in monitors list
Describe the bug
It is possible to use a balanced ip with haproxy in config.json / cm ceph-csi-config-rbd instead of provide a monitors list?
jisnardo@edxxx:~$ kubectl -n csi-rbd get cm ceph-csi-config-rbd -oyaml | grep -i config.json
default:
config.json: '[{"clusterID":"16042995-ff80-4561-8d47-70d5a80ff4ea","monitors":["192.168.0.42:3300","192.168.0.43:3300","192.168.0.44:3300"]}]'
question:
config.json: '[{"clusterID":"16042995-ff80-4561-8d47-70d5a80ff4ea","monitors":["172.16.40.117:3300"]}]'
Environment details
Image/version of Ceph CSI driver : v3.6.2
Helm chart version : v3.6.2
Kernel version : Ubuntu 20.04.4 LTS / 5.13.0-52-generic
Mounter used for mounting PVC (for cephfs its fuse or kernel. for rbd its krbd or rbd-nbd) : krbd
Kubernetes cluster version : v1.23.4
CRI: Containerd 1.5.11
Ceph cluster version : ceph version 17.2.0
Steps to reproduce
set config.json: '[{"clusterID":"16042995-ff80-4561-8d47-70d5a80ff4ea","monitors":["172.16.40.117:3300"]}]'
restart csi-rbd pods
Actual results
jisnardo@edxxxx:~$ kubectl -n csi-rbd-sc logs csi-rbd-sc-ceph-csi-rbd-provisioner-55fff46dd7-gz4cs
I0809 06:29:08.670652 1 controller.go:1337] provision "golmeda/mysql3" class "csi-rbd-sc-cephdata01": started
I0809 06:29:08.670781 1 controller.go:546] skip translation of storage class for plugin: rbd.csi.ceph.com
I0809 06:29:08.670871 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"golmeda", Name:"mysql3", UID:"8333cafe-9314-4d73-9213-4bc2ea71e150", APIVersion:"v1", ResourceVersion:"174366100", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "golmeda/mysql3"
I0809 06:29:08.681381 1 connection.go:183] GRPC call: /csi.v1.Controller/CreateVolume
I0809 06:29:08.681404 1 connection.go:184] GRPC request: {"capacity_range":{"required_bytes":1073741824},"name":"pvc-8333cafe-9314-4d73-9213-4bc2ea71e150","parameters":{"clusterID":"761d063d-748d-52c3-a29f-9aa8c22f482c","csi.storage.k8s.io/pv/name":"pvc-8333cafe-9314-4d73-9213-4bc2ea71e150","csi.storage.k8s.io/pvc/name":"mysql3","csi.storage.k8s.io/pvc/namespace":"golmeda","imageFeatures":"layering","pool":"rbd"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"ext4","mount_flags":["discard"]}},"access_mode":{"mode":1}}]}
I0809 06:29:08.682731 1 connection.go:186] GRPC response: {}
I0809 06:29:08.682768 1 connection.go:187] GRPC error: rpc error: code = Aborted desc = an operation with the given Volume ID pvc-8333cafe-9314-4d73-9213-4bc2ea71e150 already exists
I0809 06:29:08.682788 1 controller.go:788] CreateVolume failed, supports topology = false, node selected false => may reschedule = false => state = Background: rpc error: code = Aborted desc = an operation with the given Volume ID pvc-8333cafe-9314-4d73-9213-4bc2ea71e150 already exists
I0809 06:29:08.682835 1 controller.go:1082] Temporary error received, adding PVC 8333cafe-9314-4d73-9213-4bc2ea71e150 to claims in progress
W0809 06:29:08.682852 1 controller.go:934] Retrying syncing claim "8333cafe-9314-4d73-9213-4bc2ea71e150", failure 7
E0809 06:29:08.682875 1 controller.go:957] error syncing claim "8333cafe-9314-4d73-9213-4bc2ea71e150": failed to provision volume with StorageClass "csi-rbd-sc-cephdata01": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-8333cafe-9314-4d73-9213-4bc2ea71e150 already exists
I0809 06:29:08.682915 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"golmeda", Name:"mysql3", UID:"8333cafe-9314-4d73-9213-4bc2ea71e150", APIVersion:"v1", ResourceVersion:"174366100", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "csi-rbd-sc-cephdata01": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-8333cafe-9314-4d73-9213-4bc2ea71e150 already exists
Expected behavior
I0809 06:50:08.049541 1 controller.go:1337] provision "golmeda/mysql3" class "csi-rbd-sc-cephdata01": started
I0809 06:50:08.049624 1 controller.go:546] skip translation of storage class for plugin: rbd.csi.ceph.com
I0809 06:50:08.049928 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"golmeda", Name:"mysql3", UID:"e768a852-28f6-4240-a6bf-a2192ba4cb0e", APIVersion:"v1", ResourceVersion:"174379537", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "golmeda/mysql3"
I0809 06:50:08.054405 1 connection.go:183] GRPC call: /csi.v1.Controller/CreateVolume
I0809 06:50:08.054445 1 connection.go:184] GRPC request: {"capacity_range":{"required_bytes":1073741824},"name":"pvc-e768a852-28f6-4240-a6bf-a2192ba4cb0e","parameters":{"clusterID":"761d063d-748d-52c3-a29f-9aa8c22f482c","csi.storage.k8s.io/pv/name":"pvc-e768a852-28f6-4240-a6bf-a2192ba4cb0e","csi.storage.k8s.io/pvc/name":"mysql3","csi.storage.k8s.io/pvc/namespace":"golmeda","imageFeatures":"layering","pool":"rbd"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"ext4","mount_flags":["discard"]}},"access_mode":{"mode":1}}]}
I0809 06:50:08.235075 1 connection.go:186] GRPC response: {"volume":{"capacity_bytes":1073741824,"volume_context":{"clusterID":"761d063d-748d-52c3-a29f-9aa8c22f482c","imageFeatures":"layering","imageName":"csi-vol-81b2ad6c-17af-11ed-ab73-6aa4ea65d647","journalPool":"rbd","pool":"rbd"},"volume_id":"0001-0024-761d063d-748d-52c3-a29f-9aa8c22f482c-0000000000000009-81b2ad6c-17af-11ed-ab73-6aa4ea65d647"}}
I0809 06:50:08.235340 1 connection.go:187] GRPC error: <nil>
I0809 06:50:08.235353 1 controller.go:798] create volume rep: {CapacityBytes:1073741824 VolumeId:0001-0024-761d063d-748d-52c3-a29f-9aa8c22f482c-0000000000000009-81b2ad6c-17af-11ed-ab73-6aa4ea65d647 VolumeContext:map[clusterID:761d063d-748d-52c3-a29f-9aa8c22f482c imageFeatures:layering imageName:csi-vol-81b2ad6c-17af-11ed-ab73-6aa4ea65d647 journalPool:rbd pool:rbd] ContentSource:<nil> AccessibleTopology:[] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0809 06:50:08.235411 1 controller.go:867] createVolumeOperation: set annotation [volume.kubernetes.io/provisioner-deletion-secret-namespace/volume.kubernetes.io/provisioner-deletion-secret-name] on pv [pvc-e768a852-28f6-4240-a6bf-a2192ba4cb0e].
I0809 06:50:08.235440 1 controller.go:892] successfully created PV pvc-e768a852-28f6-4240-a6bf-a2192ba4cb0e for PVC mysql3 and csi volume name 0001-0024-761d063d-748d-52c3-a29f-9aa8c22f482c-0000000000000009-81b2ad6c-17af-11ed-ab73-6aa4ea65d647
I0809 06:50:08.235464 1 controller.go:908] successfully created PV {GCEPersistentDisk:nil AWSElasticBlockStore:nil HostPath:nil Glusterfs:nil NFS:nil RBD:nil ISCSI:nil Cinder:nil CephFS:nil FC:nil Flocker:nil FlexVolume:nil AzureFile:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil PortworxVolume:nil ScaleIO:nil Local:nil StorageOS:nil CSI:&CSIPersistentVolumeSource{Driver:rbd.csi.ceph.com,VolumeHandle:0001-0024-761d063d-748d-52c3-a29f-9aa8c22f482c-0000000000000009-81b2ad6c-17af-11ed-ab73-6aa4ea65d647,ReadOnly:false,FSType:ext4,VolumeAttributes:map[string]string{clusterID: 761d063d-748d-52c3-a29f-9aa8c22f482c,imageFeatures: layering,imageName: csi-vol-81b2ad6c-17af-11ed-ab73-6aa4ea65d647,journalPool: rbd,pool: rbd,storage.kubernetes.io/csiProvisionerIdentity: 1660027759774-8081-rbd.csi.ceph.com,},ControllerPublishSecretRef:nil,NodeStageSecretRef:&SecretReference{Name:csi-rbd-cephdata01-secret,Namespace:csi-rbd-sc,},NodePublishSecretRef:nil,ControllerExpandSecretRef:&SecretReference{Name:csi-rbd-cephdata01-secret,Namespace:csi-rbd-sc,},}}
I0809 06:50:08.235676 1 controller.go:1442] provision "golmeda/mysql3" class "csi-rbd-sc-cephdata01": volume "pvc-e768a852-28f6-4240-a6bf-a2192ba4cb0e" provisioned
I0809 06:50:08.235705 1 controller.go:1455] provision "golmeda/mysql3" class "csi-rbd-sc-cephdata01": succeeded
I0809 06:50:08.235718 1 volume_store.go:154] Saving volume pvc-e768a852-28f6-4240-a6bf-a2192ba4cb0e
I0809 06:50:08.242269 1 volume_store.go:157] Volume pvc-e768a852-28f6-4240-a6bf-a2192ba4cb0e saved
I0809 06:50:08.242358 1 controller.go:1069] Claim processing succeeded, removing PVC e768a852-28f6-4240-a6bf-a2192ba4cb0e from claims in progress
I0809 06:50:08.242418 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"golmeda", Name:"mysql3", UID:"e768a852-28f6-4240-a6bf-a2192ba4cb0e", APIVersion:"v1", ResourceVersion:"174379537", FieldPath:""}): type: 'Normal' reason: 'ProvisioningSucceeded' Successfully provisioned volume pvc-e768a852-28f6-4240-a6bf-a2192ba4cb0e
Logs
jisnardo@edxxxx:~$ kubectl -n golmeda describe pvc mysql3
Name: mysql3
Namespace: golmeda
StorageClass: csi-rbd-sc-cephdata01
Status: Pending
Volume:
Labels: <none>
Annotations: volume.beta.kubernetes.io/storage-provisioner: rbd.csi.ceph.com
volume.kubernetes.io/storage-provisioner: rbd.csi.ceph.com
Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode: Filesystem
Used By: <none>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning ProvisioningFailed 7m37s rbd.csi.ceph.com_csi-rbd-sc-ceph-csi-rbd-provisioner-55fff46dd7-gz4cs_0cce0f85-0f72-4ef9-b704-3db4d2c023fc failed to provision volume with StorageClass "csi-rbd-sc-cephdata01": rpc error: code = DeadlineExceeded desc = context deadline exceeded
Normal Provisioning 5m30s (x9 over 8m37s) rbd.csi.ceph.com_csi-rbd-sc-ceph-csi-rbd-provisioner-55fff46dd7-gz4cs_0cce0f85-0f72-4ef9-b704-3db4d2c023fc External provisioner is provisioning volume for claim "golmeda/mysql3"
Warning ProvisioningFailed 5m30s (x8 over 7m37s) rbd.csi.ceph.com_csi-rbd-sc-ceph-csi-rbd-provisioner-55fff46dd7-gz4cs_0cce0f85-0f72-4ef9-b704-3db4d2c023fc failed to provision volume with StorageClass "csi-rbd-sc-cephdata01": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-8333cafe-9314-4d73-9213-4bc2ea71e150 already exists
Warning ProvisioningFailed 2m44s rbd.csi.ceph.com_csi-rbd-sc-ceph-csi-rbd-provisioner-55fff46dd7-mrscd_c4231e21-de41-4fa6-8236-b6e6a01c790a failed to provision volume with StorageClass "csi-rbd-sc-cephdata01": rpc error: code = DeadlineExceeded desc = context deadline exceeded
Normal ExternalProvisioning 2m24s (x26 over 8m37s) persistentvolume-controller waiting for a volume to be created, either by external provisioner "rbd.csi.ceph.com" or manually created by system administrator
Normal Provisioning 37s (x9 over 3m44s) rbd.csi.ceph.com_csi-rbd-sc-ceph-csi-rbd-provisioner-55fff46dd7-mrscd_c4231e21-de41-4fa6-8236-b6e6a01c790a External provisioner is provisioning volume for claim "golmeda/mysql3"
Warning ProvisioningFailed 37s (x8 over 2m44s) rbd.csi.ceph.com_csi-rbd-sc-ceph-csi-rbd-provisioner-55fff46dd7-mrscd_c4231e21-de41-4fa6-8236-b6e6a01c790a failed to provision volume with StorageClass "csi-rbd-sc-cephdata01": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-8333cafe-9314-4d73-9213-4bc2ea71e150 already exists
jisnardo@edxxxx:~$ kubectl -n csi-rbd-sc logs csi-rbd-sc-ceph-csi-rbd-provisioner-55fff46dd7-mrscd -f
Defaulted container "csi-provisioner" out of: csi-provisioner, csi-resizer, csi-snapshotter, csi-attacher, csi-rbdplugin, csi-rbdplugin-controller, liveness-prometheus
I0809 06:31:37.417746 1 feature_gate.go:245] feature gates: &{map[]}
I0809 06:31:37.417806 1 csi-provisioner.go:150] Version: v3.2.1
I0809 06:31:37.417812 1 csi-provisioner.go:173] Building kube configs for running in cluster...
I0809 06:31:37.418316 1 connection.go:154] Connecting to unix:///csi/csi-provisioner.sock
I0809 06:31:38.419299 1 common.go:111] Probing CSI driver for readiness
I0809 06:31:38.419321 1 connection.go:183] GRPC call: /csi.v1.Identity/Probe
I0809 06:31:38.419327 1 connection.go:184] GRPC request: {}
I0809 06:31:38.425112 1 connection.go:186] GRPC response: {}
I0809 06:31:38.425187 1 connection.go:187] GRPC error: <nil>
I0809 06:31:38.425197 1 connection.go:183] GRPC call: /csi.v1.Identity/GetPluginInfo
I0809 06:31:38.425200 1 connection.go:184] GRPC request: {}
I0809 06:31:38.425720 1 connection.go:186] GRPC response: {"name":"rbd.csi.ceph.com","vendor_version":"v3.6.2"}
I0809 06:31:38.425779 1 connection.go:187] GRPC error: <nil>
I0809 06:31:38.425791 1 csi-provisioner.go:217] Detected CSI driver rbd.csi.ceph.com
I0809 06:31:38.425802 1 csi-provisioner.go:227] Supports migration from in-tree plugin: kubernetes.io/rbd
I0809 06:31:38.425897 1 connection.go:154] Connecting to unix:///csi/csi-provisioner.sock
I0809 06:31:38.427085 1 common.go:111] Probing CSI driver for readiness
I0809 06:31:38.427105 1 connection.go:183] GRPC call: /csi.v1.Identity/Probe
I0809 06:31:38.427111 1 connection.go:184] GRPC request: {}
I0809 06:31:38.427507 1 connection.go:186] GRPC response: {}
I0809 06:31:38.427542 1 connection.go:187] GRPC error: <nil>
I0809 06:31:38.427562 1 connection.go:183] GRPC call: /csi.v1.Identity/GetPluginCapabilities
I0809 06:31:38.427572 1 connection.go:184] GRPC request: {}
I0809 06:31:38.428350 1 connection.go:186] GRPC response: {"capabilities":[{"Type":{"Service":{"type":1}}},{"Type":{"VolumeExpansion":{"type":1}}},{"Type":{"Service":{"type":2}}}]}
I0809 06:31:38.428461 1 connection.go:187] GRPC error: <nil>
I0809 06:31:38.428471 1 connection.go:183] GRPC call: /csi.v1.Controller/ControllerGetCapabilities
I0809 06:31:38.428475 1 connection.go:184] GRPC request: {}
I0809 06:31:38.429047 1 connection.go:186] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":5}}},{"Type":{"Rpc":{"type":7}}},{"Type":{"Rpc":{"type":9}}}]}
I0809 06:31:38.429122 1 connection.go:187] GRPC error: <nil>
I0809 06:31:38.429182 1 csi-provisioner.go:289] CSI driver does not support PUBLISH_UNPUBLISH_VOLUME, not watching VolumeAttachments
I0809 06:31:38.429691 1 controller.go:732] Using saving PVs to API server in background
I0809 06:31:38.430084 1 leaderelection.go:248] attempting to acquire leader lease csi-rbd-sc/rbd-csi-ceph-com...
I0809 06:31:38.443031 1 leaderelection.go:352] lock is held by 1660026361844-8081-rbd-csi-ceph-com and has not yet expired
I0809 06:31:38.443063 1 leaderelection.go:253] failed to acquire lease csi-rbd-sc/rbd-csi-ceph-com
I0809 06:31:38.443080 1 leader_election.go:185] new leader detected, current leader: 1660026361844-8081-rbd-csi-ceph-com
I0809 06:31:49.095272 1 leaderelection.go:352] lock is held by 1660026361844-8081-rbd-csi-ceph-com and has not yet expired
I0809 06:31:49.095306 1 leaderelection.go:253] failed to acquire lease csi-rbd-sc/rbd-csi-ceph-com
I0809 06:31:58.101310 1 leaderelection.go:258] successfully acquired lease csi-rbd-sc/rbd-csi-ceph-com
I0809 06:31:58.101371 1 leader_election.go:185] new leader detected, current leader: 1660026698429-8081-rbd-csi-ceph-com
I0809 06:31:58.101390 1 leader_election.go:178] became leader, starting
I0809 06:31:58.101642 1 reflector.go:219] Starting reflector *v1.StorageClass (1h0m0s) from k8s.io/client-go/informers/factory.go:134
I0809 06:31:58.101657 1 reflector.go:255] Listing and watching *v1.StorageClass from k8s.io/client-go/informers/factory.go:134
I0809 06:31:58.101661 1 reflector.go:219] Starting reflector *v1.PersistentVolumeClaim (15m0s) from k8s.io/client-go/informers/factory.go:134
I0809 06:31:58.101672 1 reflector.go:255] Listing and watching *v1.PersistentVolumeClaim from k8s.io/client-go/informers/factory.go:134
I0809 06:31:58.111778 1 leaderelection.go:278] successfully renewed lease csi-rbd-sc/rbd-csi-ceph-com
I0809 06:31:58.202308 1 shared_informer.go:285] caches populated
I0809 06:31:58.202357 1 shared_informer.go:285] caches populated
I0809 06:31:58.202370 1 controller.go:811] Starting provisioner controller rbd.csi.ceph.com_csi-rbd-sc-ceph-csi-rbd-provisioner-55fff46dd7-mrscd_c4231e21-de41-4fa6-8236-b6e6a01c790a!
I0809 06:31:58.202405 1 clone_controller.go:66] Starting CloningProtection controller
I0809 06:31:58.202561 1 volume_store.go:97] Starting save volume queue
I0809 06:31:58.202586 1 reflector.go:219] Starting reflector *v1.StorageClass (15m0s) from sigs.k8s.io/sig-storage-lib-external-provisioner/v8/controller/controller.go:848
I0809 06:31:58.202597 1 reflector.go:255] Listing and watching *v1.StorageClass from sigs.k8s.io/sig-storage-lib-external-provisioner/v8/controller/controller.go:848
I0809 06:31:58.202670 1 clone_controller.go:82] Started CloningProtection controller
I0809 06:31:58.202755 1 reflector.go:219] Starting reflector *v1.PersistentVolume (15m0s) from sigs.k8s.io/sig-storage-lib-external-provisioner/v8/controller/controller.go:845
I0809 06:31:58.202781 1 reflector.go:255] Listing and watching *v1.PersistentVolume from sigs.k8s.io/sig-storage-lib-external-provisioner/v8/controller/controller.go:845
I0809 06:31:58.302742 1 shared_informer.go:285] caches populated
I0809 06:31:58.302947 1 controller.go:1337] provision "golmeda/mysql3" class "csi-rbd-sc-cephdata01": started
I0809 06:31:58.303018 1 controller.go:546] skip translation of storage class for plugin: rbd.csi.ceph.com
I0809 06:31:58.303168 1 controller.go:860] Started provisioner controller rbd.csi.ceph.com_csi-rbd-sc-ceph-csi-rbd-provisioner-55fff46dd7-mrscd_c4231e21-de41-4fa6-8236-b6e6a01c790a!
I0809 06:31:58.304214 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"golmeda", Name:"mysql3", UID:"8333cafe-9314-4d73-9213-4bc2ea71e150", APIVersion:"v1", ResourceVersion:"174366100", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "golmeda/mysql3"
I0809 06:31:58.307984 1 connection.go:183] GRPC call: /csi.v1.Controller/CreateVolume
I0809 06:31:58.308008 1 connection.go:184] GRPC request: {"capacity_range":{"required_bytes":1073741824},"name":"pvc-8333cafe-9314-4d73-9213-4bc2ea71e150","parameters":{"clusterID":"761d063d-748d-52c3-a29f-9aa8c22f482c","csi.storage.k8s.io/pv/name":"pvc-8333cafe-9314-4d73-9213-4bc2ea71e150","csi.storage.k8s.io/pvc/name":"mysql3","csi.storage.k8s.io/pvc/namespace":"golmeda","imageFeatures":"layering","pool":"rbd"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"ext4","mount_flags":["discard"]}},"access_mode":{"mode":1}}]}
I0809 06:32:03.134253 1 leaderelection.go:278] successfully renewed lease csi-rbd-sc/rbd-csi-ceph-com
I0809 06:32:08.150166 1 leaderelection.go:278] successfully renewed lease csi-rbd-sc/rbd-csi-ceph-com
I0809 06:32:13.175068 1 leaderelection.go:278] successfully renewed lease csi-rbd-sc/rbd-csi-ceph-com
I0809 06:32:18.190444 1 leaderelection.go:278] successfully renewed lease csi-rbd-sc/rbd-csi-ceph-com
I0809 06:32:23.213698 1 leaderelection.go:278] successfully renewed lease csi-rbd-sc/rbd-csi-ceph-com
I0809 06:32:28.227777 1 leaderelection.go:278] successfully renewed lease csi-rbd-sc/rbd-csi-ceph-com
I0809 06:32:33.254062 1 leaderelection.go:278] successfully renewed lease csi-rbd-sc/rbd-csi-ceph-com
I0809 06:32:38.269998 1 leaderelection.go:278] successfully renewed lease csi-rbd-sc/rbd-csi-ceph-com
I0809 06:32:43.290220 1 leaderelection.go:278] successfully renewed lease csi-rbd-sc/rbd-csi-ceph-com
I0809 06:32:48.306257 1 leaderelection.go:278] successfully renewed lease csi-rbd-sc/rbd-csi-ceph-com
I0809 06:32:53.327779 1 leaderelection.go:278] successfully renewed lease csi-rbd-sc/rbd-csi-ceph-com
I0809 06:32:58.308231 1 connection.go:186] GRPC response: {}
I0809 06:32:58.308354 1 connection.go:187] GRPC error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
I0809 06:32:58.308374 1 controller.go:788] CreateVolume failed, supports topology = false, node selected false => may reschedule = false => state = Background: rpc error: code = DeadlineExceeded desc = context deadline exceeded
I0809 06:32:58.308415 1 controller.go:1082] Temporary error received, adding PVC 8333cafe-9314-4d73-9213-4bc2ea71e150 to claims in progress
W0809 06:32:58.308425 1 controller.go:934] Retrying syncing claim "8333cafe-9314-4d73-9213-4bc2ea71e150", failure 0
E0809 06:32:58.308448 1 controller.go:957] error syncing claim "8333cafe-9314-4d73-9213-4bc2ea71e150": failed to provision volume with StorageClass "csi-rbd-sc-cephdata01": rpc error: code = DeadlineExceeded desc = context deadline exceeded
I0809 06:32:58.308520 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"golmeda", Name:"mysql3", UID:"8333cafe-9314-4d73-9213-4bc2ea71e150", APIVersion:"v1", ResourceVersion:"174366100", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "csi-rbd-sc-cephdata01": rpc error: code = DeadlineExceeded desc = context deadline exceeded
I0809 06:32:58.339336 1 leaderelection.go:278] successfully renewed lease csi-rbd-sc/rbd-csi-ceph-com
I0809 06:32:58.809075 1 controller.go:1337] provision "golmeda/mysql3" class "csi-rbd-sc-cephdata01": started
I0809 06:32:58.809196 1 controller.go:546] skip translation of storage class for plugin: rbd.csi.ceph.com
I0809 06:32:58.809317 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"golmeda", Name:"mysql3", UID:"8333cafe-9314-4d73-9213-4bc2ea71e150", APIVersion:"v1", ResourceVersion:"174366100", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "golmeda/mysql3"
I0809 06:32:58.817331 1 connection.go:183] GRPC call: /csi.v1.Controller/CreateVolume
I0809 06:32:58.817351 1 connection.go:184] GRPC request: {"capacity_range":{"required_bytes":1073741824},"name":"pvc-8333cafe-9314-4d73-9213-4bc2ea71e150","parameters":{"clusterID":"761d063d-748d-52c3-a29f-9aa8c22f482c","csi.storage.k8s.io/pv/name":"pvc-8333cafe-9314-4d73-9213-4bc2ea71e150","csi.storage.k8s.io/pvc/name":"mysql3","csi.storage.k8s.io/pvc/namespace":"golmeda","imageFeatures":"layering","pool":"rbd"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"ext4","mount_flags":["discard"]}},"access_mode":{"mode":1}}]}
I0809 06:32:58.818511 1 connection.go:186] GRPC response: {}
I0809 06:32:58.818549 1 connection.go:187] GRPC error: rpc error: code = Aborted desc = an operation with the given Volume ID pvc-8333cafe-9314-4d73-9213-4bc2ea71e150 already exists
I0809 06:32:58.818566 1 controller.go:788] CreateVolume failed, supports topology = false, node selected false => may reschedule = false => state = Background: rpc error: code = Aborted desc = an operation with the given Volume ID pvc-8333cafe-9314-4d73-9213-4bc2ea71e150 already exists
I0809 06:32:58.818601 1 controller.go:1082] Temporary error received, adding PVC 8333cafe-9314-4d73-9213-4bc2ea71e150 to claims in progress
W0809 06:32:58.818625 1 controller.go:934] Retrying syncing claim "8333cafe-9314-4d73-9213-4bc2ea71e150", failure 1
E0809 06:32:58.818641 1 controller.go:957] error syncing claim "8333cafe-9314-4d73-9213-4bc2ea71e150": failed to provision volume with StorageClass "csi-rbd-sc-cephdata01": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-8333cafe-9314-4d73-9213-4bc2ea71e150 already exists
I0809 06:32:58.818637 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"golmeda", Name:"mysql3", UID:"8333cafe-9314-4d73-9213-4bc2ea71e150", APIVersion:"v1", ResourceVersion:"174366100", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "csi-rbd-sc-cephdata01": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-8333cafe-9314-4d73-9213-4bc2ea71e150 already exists
I0809 06:32:59.819690 1 controller.go:1337] provision "golmeda/mysql3" class "csi-rbd-sc-cephdata01": started
I0809 06:32:59.819778 1 controller.go:546] skip translation of storage class for plugin: rbd.csi.ceph.com
I0809 06:32:59.819916 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"golmeda", Name:"mysql3", UID:"8333cafe-9314-4d73-9213-4bc2ea71e150", APIVersion:"v1", ResourceVersion:"174366100", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "golmeda/mysql3"
I0809 06:32:59.828624 1 connection.go:183] GRPC call: /csi.v1.Controller/CreateVolume
I0809 06:32:59.828657 1 connection.go:184] GRPC request: {"capacity_range":{"required_bytes":1073741824},"name":"pvc-8333cafe-9314-4d73-9213-4bc2ea71e150","parameters":{"clusterID":"761d063d-748d-52c3-a29f-9aa8c22f482c","csi.storage.k8s.io/pv/name":"pvc-8333cafe-9314-4d73-9213-4bc2ea71e150","csi.storage.k8s.io/pvc/name":"mysql3","csi.storage.k8s.io/pvc/namespace":"golmeda","imageFeatures":"layering","pool":"rbd"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"ext4","mount_flags":["discard"]}},"access_mode":{"mode":1}}]}
I0809 06:32:59.829721 1 connection.go:186] GRPC response: {}
I0809 06:32:59.829766 1 connection.go:187] GRPC error: rpc error: code = Aborted desc = an operation with the given Volume ID pvc-8333cafe-9314-4d73-9213-4bc2ea71e150 already exists
I0809 06:32:59.829783 1 controller.go:788] CreateVolume failed, supports topology = false, node selected false => may reschedule = false => state = Background: rpc error: code = Aborted desc = an operation with the given Volume ID pvc-8333cafe-9314-4d73-9213-4bc2ea71e150 already exists
I0809 06:32:59.829819 1 controller.go:1082] Temporary error received, adding PVC 8333cafe-9314-4d73-9213-4bc2ea71e150 to claims in progress
W0809 06:32:59.829847 1 controller.go:934] Retrying syncing claim "8333cafe-9314-4d73-9213-4bc2ea71e150", failure 2
E0809 06:32:59.829870 1 controller.go:957] error syncing claim "8333cafe-9314-4d73-9213-4bc2ea71e150": failed to provision volume with StorageClass "csi-rbd-sc-cephdata01": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-8333cafe-9314-4d73-9213-4bc2ea71e150 already exists
I0809 06:32:59.829892 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"golmeda", Name:"mysql3", UID:"8333cafe-9314-4d73-9213-4bc2ea71e150", APIVersion:"v1", ResourceVersion:"174366100", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "csi-rbd-sc-cephdata01": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-8333cafe-9314-4d73-9213-4bc2ea71e150 already exists
I0809 06:33:01.830946 1 controller.go:1337] provision "golmeda/mysql3" class "csi-rbd-sc-cephdata01": started
I0809 06:33:01.831024 1 controller.go:546] skip translation of storage class for plugin: rbd.csi.ceph.com
I0809 06:33:01.831136 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"golmeda", Name:"mysql3", UID:"8333cafe-9314-4d73-9213-4bc2ea71e150", APIVersion:"v1", ResourceVersion:"174366100", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "golmeda/mysql3"
I0809 06:33:01.840096 1 connection.go:183] GRPC call: /csi.v1.Controller/CreateVolume
I0809 06:33:01.840119 1 connection.go:184] GRPC request: {"capacity_range":{"required_bytes":1073741824},"name":"pvc-8333cafe-9314-4d73-9213-4bc2ea71e150","parameters":{"clusterID":"761d063d-748d-52c3-a29f-9aa8c22f482c","csi.storage.k8s.io/pv/name":"pvc-8333cafe-9314-4d73-9213-4bc2ea71e150","csi.storage.k8s.io/pvc/name":"mysql3","csi.storage.k8s.io/pvc/namespace":"golmeda","imageFeatures":"layering","pool":"rbd"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"ext4","mount_flags":["discard"]}},"access_mode":{"mode":1}}]}
I0809 06:33:01.841207 1 connection.go:186] GRPC response: {}
I0809 06:33:01.841245 1 connection.go:187] GRPC error: rpc error: code = Aborted desc = an operation with the given Volume ID pvc-8333cafe-9314-4d73-9213-4bc2ea71e150 already exists
I0809 06:33:01.841262 1 controller.go:788] CreateVolume failed, supports topology = false, node selected false => may reschedule = false => state = Background: rpc error: code = Aborted desc = an operation with the given Volume ID pvc-8333cafe-9314-4d73-9213-4bc2ea71e150 already exists
I0809 06:33:01.841301 1 controller.go:1082] Temporary error received, adding PVC 8333cafe-9314-4d73-9213-4bc2ea71e150 to claims in progress
W0809 06:33:01.841314 1 controller.go:934] Retrying syncing claim "8333cafe-9314-4d73-9213-4bc2ea71e150", failure 3
E0809 06:33:01.841333 1 controller.go:957] error syncing claim "8333cafe-9314-4d73-9213-4bc2ea71e150": failed to provision volume with StorageClass "csi-rbd-sc-cephdata01": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-8333cafe-9314-4d73-9213-4bc2ea71e150 already exists
I0809 06:33:01.841350 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"golmeda", Name:"mysql3", UID:"8333cafe-9314-4d73-9213-4bc2ea71e150", APIVersion:"v1", ResourceVersion:"174366100", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "csi-rbd-sc-cephdata01": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-8333cafe-9314-4d73-9213-4bc2ea71e150 already exists
I0809 06:33:03.353976 1 leaderelection.go:278] successfully renewed lease csi-rbd-sc/rbd-csi-ceph-com
I0809 06:33:05.842345 1 controller.go:1337] provision "golmeda/mysql3" class "csi-rbd-sc-cephdata01": started
I0809 06:33:05.842438 1 controller.go:546] skip translation of storage class for plugin: rbd.csi.ceph.com
I0809 06:33:05.842569 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"golmeda", Name:"mysql3", UID:"8333cafe-9314-4d73-9213-4bc2ea71e150", APIVersion:"v1", ResourceVersion:"174366100", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "golmeda/mysql3"
I0809 06:33:05.856834 1 connection.go:183] GRPC call: /csi.v1.Controller/CreateVolume
I0809 06:33:05.856867 1 connection.go:184] GRPC request: {"capacity_range":{"required_bytes":1073741824},"name":"pvc-8333cafe-9314-4d73-9213-4bc2ea71e150","parameters":{"clusterID":"761d063d-748d-52c3-a29f-9aa8c22f482c","csi.storage.k8s.io/pv/name":"pvc-8333cafe-9314-4d73-9213-4bc2ea71e150","csi.storage.k8s.io/pvc/name":"mysql3","csi.storage.k8s.io/pvc/namespace":"golmeda","imageFeatures":"layering","pool":"rbd"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"ext4","mount_flags":["discard"]}},"access_mode":{"mode":1}}]}
I0809 06:33:05.858053 1 connection.go:186] GRPC response: {}
I0809 06:33:05.858101 1 connection.go:187] GRPC error: rpc error: code = Aborted desc = an operation with the given Volume ID pvc-8333cafe-9314-4d73-9213-4bc2ea71e150 already exists
I0809 06:33:05.858132 1 controller.go:788] CreateVolume failed, supports topology = false, node selected false => may reschedule = false => state = Background: rpc error: code = Aborted desc = an operation with the given Volume ID pvc-8333cafe-9314-4d73-9213-4bc2ea71e150 already exists
I0809 06:33:05.858185 1 controller.go:1082] Temporary error received, adding PVC 8333cafe-9314-4d73-9213-4bc2ea71e150 to claims in progress
W0809 06:33:05.858205 1 controller.go:934] Retrying syncing claim "8333cafe-9314-4d73-9213-4bc2ea71e150", failure 4
E0809 06:33:05.858234 1 controller.go:957] error syncing claim "8333cafe-9314-4d73-9213-4bc2ea71e150": failed to provision volume with StorageClass "csi-rbd-sc-cephdata01": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-8333cafe-9314-4d73-9213-4bc2ea71e150 already exists
I0809 06:33:05.858299 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"golmeda", Name:"mysql3", UID:"8333cafe-9314-4d73-9213-4bc2ea71e150", APIVersion:"v1", ResourceVersion:"174366100", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "csi-rbd-sc-cephdata01": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-8333cafe-9314-4d73-9213-4bc2ea71e150 already exists
I0809 06:33:08.368995 1 leaderelection.go:278] successfully renewed lease csi-rbd-sc/rbd-csi-ceph-com
I0809 06:33:13.389722 1 leaderelection.go:278] successfully renewed lease csi-rbd-sc/rbd-csi-ceph-com
I0809 06:33:13.858450 1 controller.go:1337] provision "golmeda/mysql3" class "csi-rbd-sc-cephdata01": started
I0809 06:33:13.858561 1 controller.go:546] skip translation of storage class for plugin: rbd.csi.ceph.com
I0809 06:33:13.858678 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"golmeda", Name:"mysql3", UID:"8333cafe-9314-4d73-9213-4bc2ea71e150", APIVersion:"v1", ResourceVersion:"174366100", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "golmeda/mysql3"
I0809 06:33:13.863486 1 connection.go:183] GRPC call: /csi.v1.Controller/CreateVolume
I0809 06:33:13.863505 1 connection.go:184] GRPC request: {"capacity_range":{"required_bytes":1073741824},"name":"pvc-8333cafe-9314-4d73-9213-4bc2ea71e150","parameters":{"clusterID":"761d063d-748d-52c3-a29f-9aa8c22f482c","csi.storage.k8s.io/pv/name":"pvc-8333cafe-9314-4d73-9213-4bc2ea71e150","csi.storage.k8s.io/pvc/name":"mysql3","csi.storage.k8s.io/pvc/namespace":"golmeda","imageFeatures":"layering","pool":"rbd"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"ext4","mount_flags":["discard"]}},"access_mode":{"mode":1}}]}
I0809 06:33:13.864568 1 connection.go:186] GRPC response: {}
I0809 06:33:13.864603 1 connection.go:187] GRPC error: rpc error: code = Aborted desc = an operation with the given Volume ID pvc-8333cafe-9314-4d73-9213-4bc2ea71e150 already exists
I0809 06:33:13.864617 1 controller.go:788] CreateVolume failed, supports topology = false, node selected false => may reschedule = false => state = Background: rpc error: code = Aborted desc = an operation with the given Volume ID pvc-8333cafe-9314-4d73-9213-4bc2ea71e150 already exists
I0809 06:33:13.864653 1 controller.go:1082] Temporary error received, adding PVC 8333cafe-9314-4d73-9213-4bc2ea71e150 to claims in progress
W0809 06:33:13.864669 1 controller.go:934] Retrying syncing claim "8333cafe-9314-4d73-9213-4bc2ea71e150", failure 5
E0809 06:33:13.864692 1 controller.go:957] error syncing claim "8333cafe-9314-4d73-9213-4bc2ea71e150": failed to provision volume with StorageClass "csi-rbd-sc-cephdata01": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-8333cafe-9314-4d73-9213-4bc2ea71e150 already exists
I0809 06:33:13.864710 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"golmeda", Name:"mysql3", UID:"8333cafe-9314-4d73-9213-4bc2ea71e150", APIVersion:"v1", ResourceVersion:"174366100", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "csi-rbd-sc-cephdata01": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-8333cafe-9314-4d73-9213-4bc2ea71e150 already exists
I0809 06:33:18.402311 1 leaderelection.go:278] successfully renewed lease csi-rbd-sc/rbd-csi-ceph-com
I0809 06:33:23.426237 1 leaderelection.go:278] successfully renewed lease csi-rbd-sc/rbd-csi-ceph-com
I0809 06:33:28.440445 1 leaderelection.go:278] successfully renewed lease csi-rbd-sc/rbd-csi-ceph-com
I0809 06:33:29.865225 1 controller.go:1337] provision "golmeda/mysql3" class "csi-rbd-sc-cephdata01": started
I0809 06:33:29.865304 1 controller.go:546] skip translation of storage class for plugin: rbd.csi.ceph.com
I0809 06:33:29.865386 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"golmeda", Name:"mysql3", UID:"8333cafe-9314-4d73-9213-4bc2ea71e150", APIVersion:"v1", ResourceVersion:"174366100", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "golmeda/mysql3"
I0809 06:33:29.872142 1 connection.go:183] GRPC call: /csi.v1.Controller/CreateVolume
I0809 06:33:29.872161 1 connection.go:184] GRPC request: {"capacity_range":{"required_bytes":1073741824},"name":"pvc-8333cafe-9314-4d73-9213-4bc2ea71e150","parameters":{"clusterID":"761d063d-748d-52c3-a29f-9aa8c22f482c","csi.storage.k8s.io/pv/name":"pvc-8333cafe-9314-4d73-9213-4bc2ea71e150","csi.storage.k8s.io/pvc/name":"mysql3","csi.storage.k8s.io/pvc/namespace":"golmeda","imageFeatures":"layering","pool":"rbd"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"ext4","mount_flags":["discard"]}},"access_mode":{"mode":1}}]}
I0809 06:33:29.873345 1 connection.go:186] GRPC response: {}
I0809 06:33:29.873399 1 connection.go:187] GRPC error: rpc error: code = Aborted desc = an operation with the given Volume ID pvc-8333cafe-9314-4d73-9213-4bc2ea71e150 already exists
I0809 06:33:29.873418 1 controller.go:788] CreateVolume failed, supports topology = false, node selected false => may reschedule = false => state = Background: rpc error: code = Aborted desc = an operation with the given Volume ID pvc-8333cafe-9314-4d73-9213-4bc2ea71e150 already exists
I0809 06:33:29.873459 1 controller.go:1082] Temporary error received, adding PVC 8333cafe-9314-4d73-9213-4bc2ea71e150 to claims in progress
W0809 06:33:29.873477 1 controller.go:934] Retrying syncing claim "8333cafe-9314-4d73-9213-4bc2ea71e150", failure 6
E0809 06:33:29.873495 1 controller.go:957] error syncing claim "8333cafe-9314-4d73-9213-4bc2ea71e150": failed to provision volume with StorageClass "csi-rbd-sc-cephdata01": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-8333cafe-9314-4d73-9213-4bc2ea71e150 already exists
I0809 06:33:29.873512 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"golmeda", Name:"mysql3", UID:"8333cafe-9314-4d73-9213-4bc2ea71e150", APIVersion:"v1", ResourceVersion:"174366100", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "csi-rbd-sc-cephdata01": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-8333cafe-9314-4d73-9213-4bc2ea71e150 already exists
I0809 06:33:33.462447 1 leaderelection.go:278] successfully renewed lease csi-rbd-sc/rbd-csi-ceph-com
I0809 06:33:38.478559 1 leaderelection.go:278] successfully renewed lease csi-rbd-sc/rbd-csi-ceph-com
I0809 06:33:43.499776 1 leaderelection.go:278] successfully renewed lease csi-rbd-sc/rbd-csi-ceph-com
I0809 06:33:48.512481 1 leaderelection.go:278] successfully renewed lease csi-rbd-sc/rbd-csi-ceph-com
I0809 06:33:53.533888 1 leaderelection.go:278] successfully renewed lease csi-rbd-sc/rbd-csi-ceph-com
I0809 06:33:58.546946 1 leaderelection.go:278] successfully renewed lease csi-rbd-sc/rbd-csi-ceph-com
I0809 06:34:01.873903 1 controller.go:1337] provision "golmeda/mysql3" class "csi-rbd-sc-cephdata01": started
I0809 06:34:01.873990 1 controller.go:546] skip translation of storage class for plugin: rbd.csi.ceph.com
I0809 06:34:01.874128 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"golmeda", Name:"mysql3", UID:"8333cafe-9314-4d73-9213-4bc2ea71e150", APIVersion:"v1", ResourceVersion:"174366100", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "golmeda/mysql3"
I0809 06:34:01.884342 1 connection.go:183] GRPC call: /csi.v1.Controller/CreateVolume
I0809 06:34:01.884364 1 connection.go:184] GRPC request: {"capacity_range":{"required_bytes":1073741824},"name":"pvc-8333cafe-9314-4d73-9213-4bc2ea71e150","parameters":{"clusterID":"761d063d-748d-52c3-a29f-9aa8c22f482c","csi.storage.k8s.io/pv/name":"pvc-8333cafe-9314-4d73-9213-4bc2ea71e150","csi.storage.k8s.io/pvc/name":"mysql3","csi.storage.k8s.io/pvc/namespace":"golmeda","imageFeatures":"layering","pool":"rbd"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"ext4","mount_flags":["discard"]}},"access_mode":{"mode":1}}]}
I0809 06:34:01.885446 1 connection.go:186] GRPC response: {}
I0809 06:34:01.885485 1 connection.go:187] GRPC error: rpc error: code = Aborted desc = an operation with the given Volume ID pvc-8333cafe-9314-4d73-9213-4bc2ea71e150 already exists
I0809 06:34:01.885507 1 controller.go:788] CreateVolume failed, supports topology = false, node selected false => may reschedule = false => state = Background: rpc error: code = Aborted desc = an operation with the given Volume ID pvc-8333cafe-9314-4d73-9213-4bc2ea71e150 already exists
I0809 06:34:01.885554 1 controller.go:1082] Temporary error received, adding PVC 8333cafe-9314-4d73-9213-4bc2ea71e150 to claims in progress
W0809 06:34:01.885574 1 controller.go:934] Retrying syncing claim "8333cafe-9314-4d73-9213-4bc2ea71e150", failure 7
E0809 06:34:01.885593 1 controller.go:957] error syncing claim "8333cafe-9314-4d73-9213-4bc2ea71e150": failed to provision volume with StorageClass "csi-rbd-sc-cephdata01": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-8333cafe-9314-4d73-9213-4bc2ea71e150 already exists
I0809 06:34:01.885614 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"golmeda", Name:"mysql3", UID:"8333cafe-9314-4d73-9213-4bc2ea71e150", APIVersion:"v1", ResourceVersion:"174366100", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "csi-rbd-sc-cephdata01": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-8333cafe-9314-4d73-9213-4bc2ea71e150 already exists
I0809 06:34:03.561902 1 leaderelection.go:278] successfully renewed lease csi-rbd-sc/rbd-csi-ceph-com
I0809 06:34:08.577433 1 leaderelection.go:278] successfully renewed lease csi-rbd-sc/rbd-csi-ceph-com
I0809 06:34:13.598739 1 leaderelection.go:278] successfully renewed lease csi-rbd-sc/rbd-csi-ceph-com
I0809 06:34:18.617043 1 leaderelection.go:278] successfully renewed lease csi-rbd-sc/rbd-csi-ceph-com
I0809 06:34:23.641889 1 leaderelection.go:278] successfully renewed lease csi-rbd-sc/rbd-csi-ceph-com
I0809 06:34:28.657145 1 leaderelection.go:278] successfully renewed lease csi-rbd-sc/rbd-csi-ceph-com
Additional context
haproxy backend config
frontend MONS_3300
bind 172.16.40.117:3300
log-format '@cee: {"haproxy":{"cpu":{"calls":"%[cpu_calls]","ns_tot":"%[cpu_ns_tot]","ns_avg":"%[cpu_ns_avg]","lat_ns_tot":"%[lat_ns_tot]","lat_ns_avg":"%[lat_ns_avg]","pid":%pid,"time":"%Tl"},"conn":{"act":%ac,"fe":%fc,"be":%bc,"srv":%sc},"queue":{"backend":%bq,"srv":%sq},"bytes":{"uploaded":%U,"read":"%B"},"network":{"client_ip":"%ci","client_port":%cp,"frontend_ip":"%fi","frontend_port":%fp,"backend_source_ip":"%bi","backend_source_port":"%bp"},"retries":"%rc","ssl":{"version":"%sslv","ciphers":"%sslc"},"name":{"backend":"%b","frontend":"%ft","server":"%s","server_ip":"%si"},"termination_state":"%ts","time":{"timestamp":%Ts,"tw":%Tw,"tc":%Tc,"t":"%t","tt":"%Tt"},"request":{"ssl":{"client_version":"%[capture.req.hdr(0),json(utf8s)]","sni":"%[capture.req.hdr(1),json(utf8s)]"}},"response":{"status_code":%ST}}}'
default_backend cephdata01MONS_3300
mode tcp
backend cephdata01MONS_3300
mode tcp
balance source
source 172.16.40.117
server cephdata01a 172.16.41.1
# server cephdata01b 172.16.41.2:3300 check port 3300 inter 1500 rise 3 fall 3
# server cephdata01c 172.16.41.3:6789 check port 6789 inter 1500 rise 3 fall 3
Thanks in advance @Madhu-1
@jisnardo if you can reach ceph mon with HA IP, it should also work with cephcsi also.
I am not sure the Ceph protocol supports this. The Ceph MONs might (likely) send IP-addresses of the MONs to the client. The client will then connect to those IP-addresess, bypassing HAProxy again.
The cluster map contains an explanation about it, use ceph mon dump to find the hostnames/IP-addresses that a client will try to connect to.
