api-manager
api-manager copied to clipboard
Allow normal response caching for certain resources in AI APIs
Current Limitation
While the Key-Value (KV) cache has no point for resources like /chat/completions, it can be useful for resources such as listing models.
Suggested Improvement
Add the ability to configure a KV response cache for certain resources as needed.
Version
No response