polymath icon indicating copy to clipboard operation
polymath copied to clipboard

Allow configuring a shorter context to allow a shorter completion.

Open jkomoros opened this issue 3 years ago • 2 comments

The completion API sets a limit on the length of (input + output). The longer the context, the shorter the completion must be.

There's a balance between passing a lot of context to summarize and getting a long result.

It should be possible to configure that split in ask.

Originally noticed while thinking about #14

jkomoros avatar Jan 15 '23 20:01 jkomoros

  • [ ] Allow setting answer_length in sample.client.
  • [ ] Allow setting prompt_length in sample.client (sometimes you don't want to use the entire prompt; it's more expensive)

jkomoros avatar Jan 22 '23 01:01 jkomoros

Consider tiktoken to measure token length.

dalmaer avatar Feb 06 '23 04:02 dalmaer