PVC delete failed to delete 146/250 gluster volumes

[PrasadDesala]

Observed behavior

I tried deleting 250 PVCs in a sequential way using below script, for i in {1..250};do kubectl -n gcs delete pvc pvc$i;done

Below are the observations, *. All the PVCs are deleted without any issues as expected. *. Failed to delete 143 PVs, status for each PV shows as Released *. On glustercli, it failed to delete 146 gluster volumes and are in stopped state.

Expected/desired behavior

PVC delete should delete PVC, PV and teardown gluster volumes.

Details on how to reproduce (minimal and precise)

1. Create a 3 node GCS setup using vagrant.
2. Create 250 PVCs.
3. Delete all the PVCs using below script, for i in {1..250};do kubectl -n gcs delete pvc $i;done

Information about the environment:

• Glusterd2 version used (e.g. v4.1.0 or master): v6.0-dev.114.gitd51f60b
• Operating system used: Centos 7.6
• Glusterd2 compiled from sources, as a package (rpm/deb), or container:
• Using External ETCD: (yes/no, if yes ETCD version): yes; etcd Version: 3.3.8
• If container, which container image:
• Using kubernetes, openshift, or direct install:
• If kubernetes/openshift, is gluster running inside kubernetes/openshift or outside: Kubernetes

Attaching csi and gluster provisioner logs: gluster-provisioner-logs.txt csi-provisioner logs.txt