python_backend

python_backend copied to clipboard

WIP Allow Python backend to directly write Numpy arrays to SHM

1

asos-danielbunting

To support Big Endian architectures and align python backend handling of BYTES tensors with other backends, this change removes the explicit little endian flag to serialize and deserialize the length...

nnshah1

krishung5

- populating InferRequest shm_handle_ also on request loading from shared memory - fix for shm handle of PbMemory on PbTensor loading from shared memory

pziecina-nv

fix: python backend support for numpy>=2

1

Updated the PyBind11 version from `v2.10.0` to `v2.12.0`, because `v2.12` supports `numpy 2` ([Release Notes](https://github.com/pybind/pybind11/releases/tag/v2.12.0)). Before this update, if `numpy>=2.x` was installed in the same environment, performing inference requests would...

KrishnanPrash

Making Python env manager path variable configurable

2

In some restricted environments, the `tmp` directory to create python environments has to be configurable. This PR allows for the same using an environment variable `TMP_PY_ENV_PATH`. No breaking changes expected...

SachinVarghese

Trying to do `print(repr(request.inputs()[0].triton_dtype()))` throws the following exception: ``` tritonclient.utils.InferenceServerException: [StatusCode.INTERNAL] Failed to process the request(s) for model 'example', message: TypeError: Unable to convert function return value to a Python...

larmoreg

Before: ``` "infbench_summary": { "Requests": 1000, "Requests/s": 56.781948247187536, "E2E Latency(s)": 17.61123087300075, "Avg Latency(s)": 16.54096719381794, "Total Input Tokens": 127000, "Total Output Tokens": 128000, "Avg. Input Tokens / Request": 127.0, "Avg. Output...

Tabrizian

Remove shared memory key id from client error response when out-of-memory. The shared memory key can used for malicious exploitation.

yinggeh

This PR resolves the known compilation error caused by deprecated std::unary_function in Boost's container_hash implementation when building with C++17. Error Message: ``` [ 26%] Building CXX object CMakeFiles/triton-python-backend.dir/src/python_be.cc.o In file...

simonzgx