Support additional content types in post requests

[ktrapeznikov]

---

🚀 Feature

Add support to additional content-type requests such as application/octet-stream

Motivation

It will make the Litserv more flexible. It will also allow streaming binary data.

Pitch

Basically add another option to the lines here https://github.com/Lightning-AI/LitServe/blob/c8cf6b224d68bbf2006b4b20198007ead3a58fd8/src/litserve/server.py#L351

that returns request.body()

if self.request_type == Request:
        if request.headers["Content-Type"] == "application/x-www-form-urlencoded" or request.headers[
            "Content-Type"
        ].startswith("multipart/form-data"):
            payload = await request.form()
        elif request.headers["Content-Type"] == "application/octet-stream":
            payload = await request.body()
        else:
            payload = await request.json()

to support the following request

response = requests.post(API_URL, data = input_bytes, headers = {"Content-Type": "application/octet-stream"})

Alternatives

Additional context

[aniketmaurya]

hi @ktrapeznikov, could you also share the use-case/example where you would need this format?

[ktrapeznikov]

@aniketmaurya for example when using with amazon sagemaker async endpoint, an input provided as a location in an s3 bucket. AWS has some internal processors that read the file and sends it in a request. There is not much control on how the request is constructed. It basically just reads the file as bytes and sends in a body. It does not construct "multipart/form-data" request.

[PhilippeChatigny]

@ktrapeznikov Here's a workaround if you want to send bytes. I am using using safetensors as a way to serialize here but it can be any bytes.

import litserve as ls
import safetensors.numpy as packer
from fastapi import HTTPException
from fastapi.responses import Response
class BaseLitAPI(ls.LitAPI):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)

    def setup(self, device):
        self.device = device

    def decode_request(self, request): # do not put bytes as a typehint here!
        uploaded_file = request.get("file")
        if not uploaded_file:
            raise HTTPException(
                status_code=400, detail="File field is missing in form data"
            )
        file_bytes = uploaded_file.file.read()
        input_tensor = packer.load(file_bytes)["input"]
        return input_tensor

    def predict(self, x):
      return x

    def encode_response(self, output):
      response = packer.save({"output": output})
      return  Response(content=response, media_type="application/octet-stream")


if __name__ == "__main__":
    api = BaseLitAPI()
    server = ls.LitServer(api, accelerator="auto")
    server.run(port=8000)

    ## run the following client side ##
    import numpy as np
    import requests
    import safetensors.numpy as packer

    example_payload = {"input": np.random.randn(2, 10).astype(np.float32)}
    payload = packer.save(example_payload) 
    payload = {"file": payload}
    output = requests.post("http://127.0.0.1:8000/predict", files=payload)
    output = packer.load(output.content)

for litserve version 0.2.5, the content has to be a file or a JSON. Anything else by default will be parsed as JSON by default or return a 422 HTTP code if you modify the typehint of def decode_request(self, request). e.g.: def decode_request(self, request: bytes).

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.