replicate-go
replicate-go copied to clipboard
Simplify streaming code (and fix some bugs)
This simplifies the streaming code. The previous code had three parallel goroutines, this does it all in one.
I use bufio.Scanner to split response.Body on "\n\n" tokens.
There was a bug in the previous code where retries never worked: to do a retry, g.Wait() had to return an error; but if g.Wait() returned an error then ctx was canceled so the retry call to streamPrediction() was given an already-canceled context.
I've tested this locally with synthetics.