Implement graceful handling of requests/rate limits

[vaivanov95]

Validations

• [x] I believe this is a way to improve. I'll try to join the Continue Discord for questions
• [x] I'm not able to find an open issue that requests the same enhancement

Problem

Currently if during response generation an HTTP 429 code is returned from e.g. Anthropic API, the generation process halts an an error is displayed.

Solution

Potential solution

• suspend the generation for a minute and then continue
  • while the generation is suspended display a message explaining the situation: "API usage limit exceeded, retrying after (counting)"
  • display a button to trigger a retry immediately

[dabockster]

Voicing my support for implementing rate/token limit settings across the board. I'm using Gemini and OpenRouter free APIs and will often unintentionally overload them. Having the ability to manually rate limit models and customize how tokens are sent is especially critical for paid services, since most of those only have auto-billing options.

[ghost]

@tomasz-stefaniak @sestinj I'm working on it

[sestinj]

Awesome @0x23d11 let me know if we can help or you want to share your plan ahead of making a large PR

[ghost]

@sestinj I want some clarification:

• What is the scope of this issue? Do I only need to handle number of requests rate limit or are we also considering input/output token rate limits too?

• If we are handling token limits, then there are 2 situations I want clarifications about:

• • If its an error on input token rate limit, then what should be the desired way to handle this? Should we ask user to reduce it or something else?
• • If its an error on output token rate limit, then I think we need to continue from where the response was suspended.

• Can you explain the closed issue about indexing? So if it gets to rate limit, then after some retry-after time, it should continue indexing the codebase from where it got suspended? Is that right?

• How would I implement the setup where it can resume the response or indexing codebase from where it left off?

Once I get clarifications on these points, I'll start working on it. Thanks

[github-actions[bot]]

This issue hasn't been updated in 90 days and will be closed after an additional 10 days without activity. If it's still important, please leave a comment and share any new information that would help us address the issue.

[github-actions[bot]]

This issue was closed because it wasn't updated for 10 days after being marked stale. If it's still important, please reopen + comment and we'll gladly take another look!