edsl icon indicating copy to clipboard operation
edsl copied to clipboard
Published 5 months ago • verified publisher icon expectedparrot

For each inference service, add methods to learn tokens-per-minute and requests-per-minute limits

Open johnjosephhorton opened this issue 1 year ago • 1 comments

We need be able to adjust TPM/RMP limits dynamically.

If too hard, these should be user settings on coop.

johnjosephhorton avatar May 07 '24 13:05 johnjosephhorton

@zer0dss This naturally goes w/ price look-ups.

johnjosephhorton avatar Aug 15 '24 12:08 johnjosephhorton