[Feature Request]: Retry llm requests on server failures

[Ilaiwi]

Is your feature request related to a problem? Please describe.

I have been getting a lot of openai.APIConnectionError: Connection error errors while using openai and overloaded_error errors while using anthropic. This would cause the whole conversation to crash for long chain chats while a retry could probably solve the issue.

Describe the solution you'd like

Retry the request with a small backoff coffient that can be configured on the llm_configs

Additional context

No response