LitServe

LitServe copied to clipboard

% character when streaming

1

The streaming works really nicely now using the latest litgpt version from main. The only little issue is that it creates a `%` character e.g., ``` ⚡ ~/streaming python streaming_client.py...

rasbt

Feat: Add support for fastapi lifespan events

5

## 🚀 Feature Add the ability for developers to customize the FastAPI `lifespan`, `startup`, and 'shutdown` event handlers. ### Motivation Users can use this to setup logging or resource management....

robTheBuildr

feat: Update OpenAI spec to include image url in message content

8

Before submitting - [x] Was this discussed/agreed via a Github issue? (no need for typos and docs improvements) - [x] Did you read the [contributor guideline](https://github.com/Lightning-AI/pytorch-lightning/blob/main/.github/CONTRIBUTING.md), Pull Request section? -...

bhimrazy

fixes #99 * Select streaming or non-streaming loop based on client's `request["stream"]` parameter. ## Server ```py # server.py import logging import litserve as ls logging.basicConfig(level=logging.INFO) # STEP 1: DEFINE YOUR...

aniketmaurya

Add support to include images for OpenAI Chat Template

3

## 🚀 Feature Request: Add Support for Images in Chat Messages ### Description Add support for images to be included in chat messages, similar to [gpt-4o](https://platform.openai.com/docs/guides/vision). ```json { "messages": [...

bhimrazy

Investigate recycling of Pipe connections. Let’s merge it as a P0, but please add an issue to investigate recycling pipes _Originally posted by @lantiga in https://github.com/Lightning-AI/LitServe/pull/108#pullrequestreview-2073769140_

aniketmaurya

Allow LitServer to send requests to either streaming or non-streaming inference worker dynamically based on the request. cc: @lantiga

aniketmaurya

FastAPI background_task cleanup is not executed if any error is encountered. To reproduce, run this code and add a print in cleanup function. ```py import litserve as ls class SimpleLitAPI(ls.LitAPI):...

aniketmaurya

add test for failed preds and catch timeout during streaming

1

## We need to add tests for when: **- predict fails during streamed generation** -> Users will receive 200 HTTP status if fails at LitAPI side and 50x if fails...

aniketmaurya

test_timeout test is flaky

2

## 🐛 Bug ### To Reproduce Steps to reproduce the behavior: 1. Go to '...' 1. Run '....' 1. Scroll down to '....' 1. See error #### Code sample ###...

aniketmaurya