NVIDIA
bug: Nemo Guardrails failing with Vertex AI
Did you check docs and existing issues?
- [x] I have read all the NeMo-Guardrails docs
- [x] I have updated the package to the latest version before submitting this issue
- [ ] (optional) I have used the develop branch
- [x] I have searched the existing issues of NeMo-Guardrails
Python version (python --version)
3.12
Operating system/version
MacOS 15.3
NeMo-Guardrails version (if you must use a specific version and not the latest
latest
Describe the bug
When I try to use a model from vertex ai, I get the following error message:
AttributeError: property 'max_tokens' of 'VertexAI' object has no setter
Here are the package versions I am using: ` Package Version Editable project location
aiofiles 24.1.0 aiohappyeyeballs 2.4.4 aiohttp 3.10.11 aiolimiter 1.2.1 aiosignal 1.3.2 aiosqlite 0.20.0 alembic 1.14.1 alt-profanity-check 1.6.1 annotated-types 0.7.0 annoy 1.17.3 anthropic 0.37.1 anyio 4.8.0 appdirs 1.4.4 appnope 0.1.4 asttokens 3.0.0 asyncer 0.0.8 asyncio 3.4.3 asyncpg 0.30.0 attrs 25.1.0 azure-core 1.32.0 backoff 2.2.1 blis 1.2.0 boto3 1.36.14 botocore 1.36.14 cachetools 5.5.1 catalogue 2.0.10 certifi 2025.1.31 cffi 1.17.1 cfgv 3.4.0 charset-normalizer 3.4.1 click 8.1.8 cloudpathlib 0.20.0 cloudpickle 3.1.1 cohere 5.13.11 coloredlogs 15.0.1 colorlog 6.9.0 comm 0.2.2 confection 0.1.5 cryptography 44.0.0 cymem 2.0.11 dataclasses-json 0.6.7 datasets 3.2.0 debugpy 1.8.12 decorator 5.1.1 deepeval 1.6.2 defusedxml 0.7.1 Deprecated 1.2.18 dill 0.3.8 diskcache 5.6.3 distlib 0.3.9 distro 1.9.0 dnspython 2.7.0 docstring_parser 0.16 docx2txt 0.8 dspy 2.5.42 ecdsa 0.19.0 email_validator 2.2.0 en_core_web_lg 3.8.0 execnet 2.1.1 executing 2.2.0 fastapi 0.111.1 fastapi-cli 0.0.7 fastavro 1.10.0 fastembed 0.5.1 filelock 3.17.0 filetype 1.2.0 flatbuffers 25.2.10 frozenlist 1.5.0 fsspec 2024.9.0 google-ai-generativelanguage 0.6.15 google-api-core 2.24.1 google-api-python-client 2.160.0 google-auth 2.38.0 google-auth-httplib2 0.2.0 google-cloud-aiplatform 1.80.0 google-cloud-bigquery 3.29.0 google-cloud-core 2.4.1 google-cloud-resource-manager 1.14.0 google-cloud-storage 2.19.0 google-crc32c 1.6.0 google-generativeai 0.8.4 google-resumable-media 2.7.2 googleapis-common-protos 1.66.0 grpc-google-iam-v1 0.14.0 grpcio 1.63.2 grpcio-status 1.62.3 h11 0.14.0 httpcore 1.0.7 httplib2 0.22.0 httptools 0.6.4 httpx 0.28.1 httpx-sse 0.4.0 huggingface-hub 0.28.1 humanfriendly 10.0 identify 2.6.6 idna 3.10 importlib-metadata 7.0.0 iniconfig 2.0.0 ipykernel 6.29.5 ipython 8.32.0 jedi 0.19.2 Jinja2 3.1.5 jiter 0.8.2 jmespath 1.0.1 joblib 1.4.2 json_repair 0.35.0 jsonpatch 1.33 jsonpointer 3.0.0 jsonschema 4.23.0 jsonschema-specifications 2024.10.1 jupyter_client 8.6.3 jupyter_core 5.7.2 kanzy-agent 0.1.0 /Users/emirarditi/PycharmProjects/kanzy-agent langchain 0.3.17 langchain-anthropic 0.2.4 langchain-aws 0.2.12 langchain-community 0.3.16 langchain-core 0.3.33 langchain-google-genai 2.0.9 langchain-google-vertexai 2.0.13 langchain-milvus 0.1.8 langchain-openai 0.2.14 langchain-pinecone 0.2.2 langchain-tests 0.3.10 langchain-text-splitters 0.3.5 langchain-voyageai 0.1.3 langcodes 3.5.0 langgraph 0.2.69 langgraph-checkpoint 2.0.10 langgraph-checkpoint-sqlite 2.0.3 langgraph-sdk 0.1.51 langsmith 0.3.6 language_data 1.3.0 lark 1.1.9 litellm 1.51.0 loguru 0.7.3 magicattr 0.1.6 Mako 1.3.9 marisa-trie 1.2.1 markdown-it-py 3.0.0 MarkupSafe 3.0.2 marshmallow 3.26.1 matplotlib-inline 0.1.7 mdurl 0.1.2 milvus-lite 2.4.11 mmh3 4.1.0 mpmath 1.3.0 msgpack 1.1.0 multidict 6.1.0 multiprocess 0.70.16 murmurhash 1.0.12 mypy-extensions 1.0.0 nemoguardrails 0.11.0 nest-asyncio 1.6.0 nodeenv 1.9.1 numpy 1.26.4 onnxruntime 1.20.1 openai 1.61.1 opentelemetry-api 1.24.0 opentelemetry-exporter-otlp-proto-common 1.24.0 opentelemetry-exporter-otlp-proto-grpc 1.24.0 opentelemetry-proto 1.24.0 opentelemetry-sdk 1.24.0 opentelemetry-semantic-conventions 0.45b0 optuna 4.2.0 orjson 3.10.15 packaging 24.2 pandas 2.2.3 parso 0.8.4 pexpect 4.9.0 phonenumbers 8.13.54 pillow 10.4.0 pinecone 5.4.2 pinecone-plugin-inference 3.1.0 pinecone-plugin-interface 0.0.7 pip 24.3.1 platformdirs 4.3.6 pluggy 1.5.0 portalocker 3.1.1 pre-commit 3.8.0 preshed 3.0.9 presidio_analyzer 2.2.357 presidio_anonymizer 2.2.357 prompt_toolkit 3.0.50 propcache 0.2.1 proto-plus 1.26.0 protobuf 4.25.6 psutil 6.1.1 psycopg2 2.9.10 ptyprocess 0.7.0 pure_eval 0.2.3 py_rust_stemmers 0.1.3 pyarrow 19.0.0 pyasn1 0.6.1 pyasn1_modules 0.4.1 pycparser 2.22 pycryptodome 3.21.0 pydantic 2.11.0a1 pydantic_core 2.28.0 pydantic-settings 2.7.1 Pygments 2.19.1 pymilvus 2.5.4 pyparsing 3.2.1 pytest 8.3.4 pytest-asyncio 0.25.3 pytest-order 1.3.0 pytest-repeat 0.9.3 pytest-socket 0.7.0 pytest-xdist 3.6.1 python-dateutil 2.9.0.post0 python-dotenv 1.0.1 python-jose 3.3.0 python-multipart 0.0.20 pytz 2025.1 PyYAML 6.0.2 pyzmq 26.2.1 ragas 0.2.13 redis 5.2.1 referencing 0.36.2 regex 2024.11.6 requests 2.32.3 requests-file 2.1.0 requests-toolbelt 1.0.0 rich 13.9.4 rich-toolkit 0.13.2 rpds-py 0.22.3 rsa 4.9 s3transfer 0.11.2 scikit-learn 1.6.1 scipy 1.15.1 sentry-sdk 2.20.0 setuptools 75.8.0 shapely 2.0.7 shellingham 1.5.4 simpleeval 1.0.3 six 1.17.0 smart-open 7.1.0 sniffio 1.3.1 spacy 3.8.4 spacy-legacy 3.0.12 spacy-loggers 1.0.5 SQLAlchemy 2.0.37 sqlglot 25.34.1 srsly 2.5.1 stack-data 0.6.3 starlette 0.37.2 sympy 1.13.3 syrupy 4.8.1 tabulate 0.9.0 tavily-python 0.5.0 tenacity 8.4.2 thinc 8.3.4 threadpoolctl 3.5.0 tiktoken 0.8.0 tldextract 5.1.3 tokenizers 0.21.0 tornado 6.4.2 tqdm 4.67.1 traitlets 5.14.3 typer 0.15.1 types-requests 2.32.0.20241016 typing_extensions 4.12.2 typing-inspect 0.9.0 tzdata 2025.1 ujson 5.10.0 uritemplate 4.1.1 urllib3 2.3.0 uvicorn 0.29.0 uvloop 0.21.0 virtualenv 20.29.1 voyageai 0.2.4 wasabi 1.1.3 watchdog 6.0.0 watchfiles 1.0.4 wcwidth 0.2.13 weasel 0.4.1 websockets 14.2 wrapt 1.17.2 xxhash 3.5.0 yarl 1.18.3 zipp 3.21.0 zstandard 0.23.0
`
Steps To Reproduce
Set config to: models:
- type: main engine: vertexai model: gemini-2.0-flash-001
Expected Behavior
For it to work and produce rail output
Actual Behavior
The following error:
AttributeError: property 'max_tokens' of 'VertexAI' object has no setter
max_tokens is a read-only property in langchain-google-vertexai 2.0.13, not sure why there is no setter. See discussion linked above. The solution should be to also add a setter in langchain-google-vertexai on my side - it does not make sense to add a patch.
This is a nice bug that appeared with the new released langchain-google-vertexai 2.0.13: they introduced max_tokens as a property, as this was missing until version 2.0.13. However, it's only a read-only property, so we cannot modify it here:
https://github.com/NVIDIA/NeMo-Guardrails/blob/1d9f278dd9dbe42025669d7d79a7b885d187ee6f/nemoguardrails/llm/params.py#L44
@Pouyanpi , should we patch this on our side? A simple try-catch on AttributeError would fix this for any Langchain LLM provider,
Thank you, @trebedea. I noticed that vertexai originally used max_output_tokens aliased as max_tokens without defining an explicit max_tokens property. This meant that in older versions, passing max_tokens worked without raising errors, even though it wasn’t clearly exposed (we were passing max_tokens silently).
Having a property named max_tokens can be confusing. I think langchain-google-vertexai should properly support a setter as you mentioned. We can possibly resolve this by patching both the getter and setter (if the property didn't exist fro previous versions). I think, using a try-catch on AttributeError would simply ignore the issue( it would fail to set the attribute without any notification, leaving developers unaware that the intended update wasn’t applied).
I can see two options as you suggested:
- Patch vertexai similarly to what we did for ChatNVIDIA, by implementing both a setter and getter.
- Open an issue on langchain-google-vertexai, acknowledging that we cannot support versions prior to 2.0.14.
This lack of unified parameters across different providers is indeed problematic.
let me know if you have something else in mind.
...
@property
def max_tokens(self) -> Optional[int]:
return self.max_output_tokens
@max_tokens.setter
def max_tokens(self, value: Optional[int]) -> None:
self.max_output_tokens = value
this should be fixed in 0.17.0 release of nemoguardrails. We can test using the head of develop branch.