weaviate
ssl error when connecting self-signed url (happned right after client >= 4.7.0)
Our weaviate db exposed with nginx ingress, site https://myweb.com is a company self-signed URL.
To access it, it use following piece of code and was working fine till 4.6.7.
os.environ["GRPC_DEFAULT_SSL_ROOTS_FILE_PATH"] = "/home/jovyan/cert.crt" os.environ["SSL_CERT_FILE"] = "/home/jovyan/cert.crt"
client = weaviate.connect_to_custom( http_host="myweb.com", http_port=443, http_secure=True, grpc_host="myweb.com", grpc_port=443, grpc_secure=True, auth_credentials=weaviate.auth.AuthApiKey(id_token), additional_config=weaviate.config.AdditionalConfig( startup_period=10, timeout=(5, 15) ), ) client.connect()
When we use 4.7.0, it breaks, it always saying:
File /opt/conda/lib/python3.11/site-packages/anyio/streams/tls.py:133, in TLSStream._call_sslobject_method(self, func, *args)
131 while True:
132 try:
--> 133 result = func(*args)
134 except ssl.SSLWantReadError:
135 try:
136 # Flush any pending writes first
File /opt/conda/lib/python3.11/ssl.py:979, in SSLObject.do_handshake(self)
977 def do_handshake(self):
978 """Start the SSL/TLS handshake."""
--> 979 self._sslobj.do_handshake()
SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)
We have to use fixed version 4.6.7, can you help check here?
Hi @Alansyf, it would seem that there is a subtle change in how the underlying grpc C implementation handles SSL certificates between the sync (grpc) and async (grpc.aio) modules
Now that the connection used by the client is fully async, this issue is raising its head!
I think the general solution here, from researching others' issues, is to allow users to specify the SSL certs in their client configuration. The client will then handle piping these to the respective httpx and grpc.aio connections. I will look into this in the coming days and release a patch fix once complete! Thanks for the report 😁
Hi @Alansyf, sorry for letting this go stale! We recently refactored the sync client to use fully sync connections under-the-hood once more (httpx and grpc) so this issue may have been fixed as a result. Could you try with the latest version of the client to see if it alleviates your issue?
Hi,
I think I have the same problem or at least a similar one as the creator of this issue.
I have a weaviate service deployed inside a k8s cluster which has it's own CA (caused by using an internal fqdn + ip which I can't get valid let's encrypt ca for). Now if I run the following script I get similar issues as mentioned above:
import weaviate
from weaviate.classes.init import Auth
http_host = "weaviate.example.com"
grpc_host = "weaviate-grpc.example.com"
weaviate_api_key = "SuperDooperSecure"
client = weaviate.connect_to_custom(
http_host=http_host,
http_port=443,
http_secure=True,
grpc_host=grpc_host,
grpc_port=443,
grpc_secure=True,
auth_credentials=Auth.api_key(weaviate_api_key),
)
print(f"ready: {client.is_ready()}")
meta_info = client.get_meta()
print(meta_info)
client.close()
I've tried to come up with a solution to fix the problem of how to set up a CUSTOM CA (look below what I was executing before any other code is running). This solution is working but I wouldn't recommend it because this will overwrite or append indefinitely the custom ca to certifi's ca.
import os
if os.environ['CUSTOM_CA']:
os.environ['GRPC_DEFAULT_SSL_ROOTS_FILE_PATH'] = os.environ['CUSTOM_CA']
import certifi
with open(certifi.where(), "w") as certifi_ca: # "a" for append instead of "w"
with open(os.environ['CUSTOM_CA'], "r") as custom_ca:
certifi_ca.writelines(custom_ca.readlines())
Findings
For gRPC exists a simple solution using a custom CA. You simply have to set the GRPC_DEFAULT_SSL_ROOTS_FILE_PATH I've validated this with the quick demo script and setting the environment value to the ca.pem file.
For HTTP there is no easy solution, you could overwrite the cacert.pem inside the certifi package which is, based on this documentation, an anti pattern and shouldn't be considered. Another (better) solution is to configure the ca via the http library (eg. requests or httpx). requests implemented this behaviour in itself. httpx requires the application to configure the custom ca itself. This fact is mentioned in the httpx documentation. Based on the documentation it implies that the application has to create a client context in which the custom ca is configured. Downside here is the easy usability via httpx.get(...) might not work anymore and further changes in the code has to be made.
Hi,
I have been struggling for quite some time now with this issue as well but finally managed to get it to work now with my organizations custom CA with a weaviate instance hosted in a k8s cluster. Code;
import weaviate
import os
from weaviate.config import AdditionalConfig
cert_path = "/path/to/our_custom_ca.crt"
os.environ['SSL_CERT_FILE'] = cert_path
os.environ['GRPC_DEFAULT_SSL_ROOTS_FILE_PATH'] = cert_path
key = "mykey"
auth = weaviate.auth.Auth.api_key(
key
)
http_host = "weaviate.server.com"
grpc_host = "weaviate-grpc.server.com"
with weaviate.connect_to_custom(
http_host=http_host,
http_port=443,
http_secure=True,
grpc_host=grpc_host,
grpc_port=443,
grpc_secure=True,
auth_credentials=auth,
additional_config=AdditionalConfig(trust_env=True)
) as client:
print(client.is_ready())
Using httpx~=0.28.1 & weaviate-client~=4.17.0. The key different was passing the additional_config AdditionalConfig(trust_env=True), which resulted in trust_env being passed down to the httpx config & thus allowing us to rely on the environment variables again (as seen in the code here).
I appreciate your comment @TheMrPoseidon - it made me give this another shot. Hopefully, this helps someone else out too. 👍
Hey - our documentation has a section about custom SSL certificates: https://docs.weaviate.io/weaviate/client-libraries/python/notes-best-practices#using-custom-ssl-certificates - sorry that we didn't update here
