Implement Tools Config Caching for Anthropic Claude

[chezsmithy]

Validations

• [x] I believe this is a way to improve. I'll try to join the Continue Discord for questions
• [x] I'm not able to find an open issue that requests the same enhancement

Problem

I would like to add prompt caching for the tools configuration for Anthropic, AWS Bedrock and Vertex hosted Claude models. Tools config is highly repeatable and caching this configuration limits the number of tokens used. I've validated this with my local configuration using AWS Bedrock with Claude 3.7.

For AWS Bedrock and Claude 3.7 see this page for more information: https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-caching.html

Solution

I've identified through local testing that it's possible to add caching support for tools config. This allows the config to enter the prompt cache and then significantly lowers the number of tokens consumed on each subsequent call to the LLM.

I've identified that AWS Bedrock, Anthropic and Vertex seem to all support this feature tho in slightly different ways.

Here is what I would propose as PRs.

1. Create a new configuration item to enable toolsConfigCache under the cacheBehavior key.
2. Using that key have contributors with access to each back end independently add the caching support as a separate PR with updated tests.

Implementation Questions

1. Should I implement the configuration changes in JSON and YAML?
2. Should the configuration only be enabled for certain models and providers in a specific way?

[chezsmithy]

Noting I'm willing to contribute a PR to first implement the configuration changes.

Then once that is approved, I will submit a PR for AWS Bedrock Claude 3.7.

[chezsmithy]

Once the Cache Behavior merges then I'll open a PR for the Bedrock provider and Claude model.

[chezsmithy]

#5371 This is resolved.

[chezsmithy]

@Patrick-Erichsen we likely want to do a scan of the other providers implementating caching and make sure they are using the new combined promptCaching settting and start to deprecate the individual settings.

[github-actions[bot]]

This issue hasn't been updated in 90 days and will be closed after an additional 10 days without activity. If it's still important, please leave a comment and share any new information that would help us address the issue.

[github-actions[bot]]

This issue was closed because it wasn't updated for 10 days after being marked stale. If it's still important, please reopen + comment and we'll gladly take another look!