LitServe
LitServe copied to clipboard
add test for failed preds and catch timeout during streaming
We need to add tests for when:
- predict fails during streamed generation -> Users will receive 200 HTTP status if fails at LitAPI side and 50x if fails at endpoint side. - ~a timeout happens during streaming~ -> we can't do this since, FastAPI will just stream the output till timeout instead of throwing an error. - test end-to-end streaming: Done
Originally posted by @lantiga in https://github.com/Lightning-AI/litserve/pull/37#discussion_r1566478060
I think we should close it now @lantiga