replicate-go icon indicating copy to clipboard operation
replicate-go copied to clipboard

Simplify streaming code (and fix some bugs)

Open philandstuff opened this issue 1 year ago • 0 comments

This simplifies the streaming code. The previous code had three parallel goroutines, this does it all in one.

I use bufio.Scanner to split response.Body on "\n\n" tokens.

There was a bug in the previous code where retries never worked: to do a retry, g.Wait() had to return an error; but if g.Wait() returned an error then ctx was canceled so the retry call to streamPrediction() was given an already-canceled context.

I've tested this locally with synthetics.

philandstuff avatar Oct 03 '24 00:10 philandstuff