GenAIExamples icon indicating copy to clipboard operation
GenAIExamples copied to clipboard
Published 2 months ago • verified publisher icon opea-project

Gateway guardrail

Open kevinintel opened this issue 1 year ago • 2 comments

kevinintel avatar Aug 12 '24 07:08 kevinintel

will assign to Zhihao

kevinintel avatar Aug 12 '24 07:08 kevinintel

image

We can see from the similar project that the guardrails is outside of the inferencing pipeline.

Currently in OPEA, we have the guardrails microservice inside the pipeline, but we also need the guardrails outside the pipeline, or we say, in the gateway side, which can not only judge the content safety from either the pipeline request or response, but also check the relevance between the pipeline request and response.

We now have the gateway guardrails PoC on Envoy, but we do not have gateway itself, neither in GenAIExamples nor GenAIInfra. We have to decide which gateway should be used, Nginx, Envoy or APISIX, what the API should be, and finally the implementation.

See also https://github.com/opea-project/GenAIInfra/issues/197.

zhxie avatar Aug 15 '24 03:08 zhxie