Discover Cassandra nodes via Kubernetes DNS during replication

[c-w]

Currently, the helm chart that sets up Cassandra has nodes discovering each other by IP during data replication. In rare situations we've seen this cause issues if Cassandra and Kubernetes get out of sync as to what are the correct IP addresses for the Cassandra workers. To prevent this issue and shield Cassandra from IP changes, we want to front each Cassandra node with a DNS name of a kubernetes service. @xtophs is the expert in this area for further questions.

[c-w]

Error screenshots from yet another outage caused by this:

[c-w]

Tried to fix the issue with kubectl -n cassandra exec -it cassandra-cluster-cassan-0 -- nodetool removenode 7d0841ec-09fc-4c5c-b67b-412d0d8a0afb and kubectl -n cassandra exec -it cassandra-cluster-cassan-0 -- nodetool removenode 860d09c0-1571-4771-9574-6d537fee57c1.