NeMo-Guardrails icon indicating copy to clipboard operation
NeMo-Guardrails copied to clipboard

bug: Guardrail server failing when streaming is enabled

Open swayam-singhal opened this issue 6 months ago • 0 comments

Did you check docs and existing issues?

  • [x] I have read all the NeMo-Guardrails docs
  • [x] I have updated the package to the latest version before submitting this issue
  • [ ] (optional) I have used the develop branch
  • [x] I have searched the existing issues of NeMo-Guardrails

Python version (python --version)

Python 3.12.5

Operating system/version

MacOS 15.3.2

NeMo-Guardrails version (if you must use a specific version and not the latest

No response

Describe the bug

models:

  • type: main engine: openai model: gpt-4.1-2025-04-14 parameters: temperature: 0.2 stream: true

for this config trying to work with the nemoguardrails server with stream: true it is failing with the issue below:

File "/Users/Swayam/Documents/Projects/pandemos/client/.venv/lib/python3.12/site-packages/nemoguardrails/library/self_check/input_check/actions.py", line 72, in self_check_input response = await llm_call(llm, prompt, stop=stop) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/Swayam/Documents/Projects/pandemos/client/.venv/lib/python3.12/site-packages/nemoguardrails/actions/llm/utils.py", line 122, in llm_call raise LLMCallException(e) nemoguardrails.actions.llm.utils.LLMCallException: LLM Call Exception: 'AsyncStream' object has no attribute 'model_dump'

Steps To Reproduce

  1. Create an OpenAI config with stream true
  2. Start the nemoguardrails server
  3. Try to ask any query or use a custom openai client with local base url to ask AsyncOpenAI streaming queries.

Expected Behavior

  1. Streaming response should be returned neatly.

Actual Behavior

  1. Internal server error occurs.

swayam-singhal avatar Aug 07 '25 07:08 swayam-singhal