config-reloader Restarts Multiple Times on Boot

[silvamerica]

I've been investigating why the config-reloader pod occasionally restarts multiple times while booting, and I've narrowed it down to this line. I'm seeing the following in the logs:

Get \"http://localhost:19090/-/ready\": dial tcp 127.0.0.1:19090: connect: connection refused

Per the comment on line 73, it seems like the intent would be to continue polling in this situation, but instead the process exits and has to be restarted.

[pintohutch]

Hi @silvamerica,

Thanks for pointing this out!

The thought behind not tolerating the error was that it would be a non-recoverable issue with the http client or something. However, as it turns out, that's not the case - as you have highlighted.

Open PR at #474.

[mathe-matician]

@pintohutch to clarify, will we lose any metrics being collected on that node if we encounter this crash? Or is this just isolated to the config-reloader sidecar?

[pintohutch]

Hey @mathe-matician - you will not lose metrics. This is just the config-reloader restarting because prometheus isn't ready to load config yet (presumably due to startup delay).