api-manager icon indicating copy to clipboard operation
api-manager copied to clipboard

Allow normal response caching for certain resources in AI APIs

Open Arshardh opened this issue 3 months ago • 0 comments

Current Limitation

While the Key-Value (KV) cache has no point for resources like /chat/completions, it can be useful for resources such as listing models.

Suggested Improvement

Add the ability to configure a KV response cache for certain resources as needed.

Version

No response

Arshardh avatar Oct 27 '25 19:10 Arshardh