LitServe icon indicating copy to clipboard operation
LitServe copied to clipboard

add test for failed preds and catch timeout during streaming

Open aniketmaurya opened this issue 1 year ago • 1 comments

We need to add tests for when:

- predict fails during streamed generation -> Users will receive 200 HTTP status if fails at LitAPI side and 50x if fails at endpoint side. - ~a timeout happens during streaming~ -> we can't do this since, FastAPI will just stream the output till timeout instead of throwing an error. - test end-to-end streaming: Done

Originally posted by @lantiga in https://github.com/Lightning-AI/litserve/pull/37#discussion_r1566478060

aniketmaurya avatar Apr 16 '24 15:04 aniketmaurya

I think we should close it now @lantiga

aniketmaurya avatar May 14 '24 16:05 aniketmaurya