Kishore Nallan
Kishore Nallan
Was this node part of another cluster previously? See the last IP in the log line: `10.0.0.5` -- seems like a new IP. Can you try stopping all the nodes,...
Is there anything that Typesense can do to help with this? If the DNS returns stale IP, it's difficult to detect that and work around that. We've never faced any...
Yes, Typesense already resolves the DNS every 10 seconds, so it seems like somehow the DNS cache is always stale. The sidecar approach should certainly work, as that would mimic...
@seeruk Thank you for putting this together 🙏 . I think this is certainly the best way to handle node rotations. I will see if we can get this integrated...
If we can merge the changes back to https://github.com/typesense/typesense-kubernetes that will be great!
Yes, that sounds perfect 👍
One thing to keep in mind is the speed of rotation: did you ensure that the nodes actually caught up after each rotation? If you rotate too fast, cluster loses...
> This might be tied to the readiness check responding as ready too early in that case What do you use for the readiness check? The `/health` should return healthy...
That still sounds a bit concerning because with the swap from DNS to IP address, not sure how Typesense will handle it. Can you use maybe a port based check...
Thanks for creating this issue @obottek. We will add this to our backlog when we tackle the next round of relevancy improvements.