grpc
xDS with TrafficDirector and spiffe: CertificateException: no name matching foo.bar found
grpc_java version 1.60.0
Hello,
I'm trying to get traffic director to provide security configuration to a client but I am encountering a failure in checkServerTrusted of XdsX509TrustManager https://github.com/grpc/grpc-java/blob/v1.60.0/xds/src/main/java/io/grpc/xds/internal/security/trust/XdsX509TrustManager.java#L244 because the target hostname does not match the SAN (spiffe id) in the server cert. I am wondering if the problem lies with the config from TD or with the client.
here is a snippet of the java code, written using the xds example in this repo:
var credentials = XdsChannelCredentials.create(InsecureChannelCredentials.create());
var channel = Grpc.newChannelBuilder("xds:///foo.bar:8443", credentials);
var blockingStub = FooGrpc.newBlockingStub(channel);
blockingStub.sayHello("world")
stack trace:
Caused by java.security.cert.CertificateException: No name matching foo.bar found
at java.base/sun.security.util.HostnameChecker.matchDNS(HostnameChecker.java:229)
at java.base/sun.security.util.HostnameChecker.match(HostnameChecker.java:103)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:457)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:417)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:291)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144)
at io.grpc.xds.internal.security.trust.XdsX509TrustManager.checkServerTrusted(XdsX509TrustManager.java:244)
at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1296)
and this is xds config traffic director is giving the client:
{
"config": [
{
"node": {
"id": "projects/1337/networks/mesh:foo-mesh/nodes/1338",
"metadata": {
"INSTANCE_IP": "0.0.0.0",
"TRAFFICDIRECTOR_GCP_PROJECT_NUMBER": "1337",
"TRAFFICDIRECTOR_MESH_NAME": "foo-mesh",
"TRAFFICDIRECTOR_NETWORK_NAME": "foo-network",
"XDS_STREAM_TYPE": "ADS"
},
"locality": {
"zone": "ib"
},
"userAgentName": "gRPC Java",
"userAgentVersion": ""
},
"xdsConfig": [
{
"status": "SYNCED",
"clusterConfig": {
"dynamicActiveClusters": [
{
"cluster": {
"@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
"name": "cloud-internal-istio:cloud_mp_7555213123_1231234123",
"altStatName:": "/projects/1339/global/backendServices/foo-bar-td-backendsvc",
"type": "EDS",
"edsClusterConfig": {
"edsConfig": {
"ads": {},
"initialFetchTimeout": "15s",
"resourceApiVersion": "V3"
}
},
"connectTimeout": "30s",
"http2ProtocolOptions": {
"maxConcurrentStreams": 100
},
"transportSocket": {
"name": "envoy.transport_sockets.tls",
"typedConfig": {
"@type": "",
"commonTlsContext": {
"tlsCertificateSdsSecretConfigs": [
{
"name": "tls_sds",
"sdsConfig": {
"path": "/etc/envoy/tls_certificate_context_sds_secret.yaml"
}
}
],
"tlsCertificateCertificateProviderInstance": {
"instanceName": "google_cloud_private_spiffe",
"certificateName": "DEFAULT"
},
"combinedValidationContext": {
"defaultValidationContext": {
"matchSubjectAltNames": [
{
"exact": "spiffe://foo.svc.id.goog/ns/bar/sa/foo"
}
]
},
"validationContextSdsSecretConfig": {
"name": "validation_context_sds",
"sdsConfig": {
"path": "/etc/envoy/tls_certificate_context_sds_secret.yaml"
}
},
"validationContextCertificateProviderInstance": {
"instanceName": "google_cloud_private_spiffe",
"certificateName": "ROOTCA"
}
},
"alpnProtocols": [
"h2"
]
}
}
},
"metadata": {
"filterMetadata": {
"com.google.trafficdirector": {
"backend_service_name": "foo-bar-td-backendsvc",
"backend_service_project_number": 1339
}
}
}
}
}
]
}
},
{
"status": "SYNCED",
"listenerConfig": {
"dynamicListeners": [
{
"activeState": {
"listener": {
"@type": "type.googleapis.com/envoy.config.listener.v3.Listener",
"name": "foo.bar:8443",
"apiListener": {
"apiListener": {
"@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
"statPrefix": "trafficdirector",
"rds": {
"configSource": {
"ads": {},
"resourceApiVersion": "V3"
},
"routeConfigName": "projects/1339/locations/global/grpcRoutes/foo-bar-td-grpcroute_foo_bar:8443"
},
"httpFilters": [
{
"name": "envoy.filters.http.fault",
"typedConfig": {
"@type": "type.googleapis.com/envoy.extensions.filters.http.fault.v3.HTTPFault"
}
},
{
"name": "envoy.filters.http.router",
"typedConfig": {
"@type": "type.googleapis.com/envoy.extensions.filters.http.router.v3.Router",
"suppressEnvoyHeaders": true
}
}
]
}
}
}
}
}
]
}
},
{
"status": "SYNCED",
"routeConfig": {
"dynamicRouteConfigs": {
"routeConfig": {
"@type": "type.googleapis.com/envoy.config.route.v3.RouteConfiguration",
"name": "projects/1339/locations/global/grpcRoutes/foo-bar-td-grpcroute_foo_bar:8443",
"virtualHosts": [
{
"domains": [
"foo.bar:8443"
],
"routes": [
{
"match": {
"prefix": ""
},
"route": {
"cluster": "cloud-internal-istio:cloud_mp_7555213123_1231234123"
}
}
]
}
]
}
}
}
}
]
}
]
}
I should mention also TD is being configured via a NetworkSecurityClientTLSPolicy attached to a ComputeBackendService as documented here: https://cloud.google.com/traffic-director/docs/security-overview#:~:text=A%20client%20TLS%20policy%20lets,a%20global%20backend%20service%20resource.
I've tried adding the spiffe id as an subjectAltName to the computebackend service which does appear in the xds config under defaultValidationContext but iiuc that isnt used until verifySubjectAltNameInChain in https://github.com/grpc/grpc-java/blob/v1.60.0/xds/src/main/java/io/grpc/xds/internal/security/trust/XdsX509TrustManager.java#L245
Thanks
From a brief glance, spiffe id is incorrect. Normally it would be spiffe://{WORKLOAD_POOL}/ns/{NAMESPACE}/sa/{KUBERNETES_SERVICE_ACCOUNT}, where WORKLOAD_POOL is "{PROJECT_ID}.svc.id.goog". Not sure if the first foo in spiffe://foo.svc.id.goog/ns/bar/sa/foo is the same as the second one, but could you please double check it's a project id?
From a brief glance, spiffe id is incorrect. Normally it would be
spiffe://{WORKLOAD_POOL}/ns/{NAMESPACE}/sa/{KUBERNETES_SERVICE_ACCOUNT}, where WORKLOAD_POOL is"{PROJECT_ID}.svc.id.goog". Not sure if the first foo inspiffe://foo.svc.id.goog/ns/bar/sa/foois the same as the second one, but could you please double check it's a project id?
sorry thats part is on me removing some information before posting.. the actual spiffe id is correct but its that not being equal to foo.bar thats throwing the exception
Also, that CSDS dump doesn't show any endpoints. It should be something like
"endpoints": [{
"locality": {
"subZone": "ib:us-central1-a_2827676496526826176_neg"
},
"lbEndpoints": [{
"endpoint": {
"address": {
"socketAddress": {
"address": "10.12.0.189",
"portValue": 8080
}
}
},
"healthStatus": "HEALTHY"
}],
"loadBalancingWeight": 1000000
}]
I'm attaching a log file with xDS responses received by grpc-java 1.60.x xDS client setup for spiffe TLS: downloaded-logs-20240110-104310.csv. Hope this helps while we're looking into this.
Here's the TD setup we did for the attached client log: setup.log. However, It's not GCE, but k8s-to-k8s.
Which Java version are you using? I'm suspicious that it may matter.
It seems weird to me that foo.bar is getting checked from the JDK's checkServerTrusted(). The gRPC code is at least trying to disable that check. The null check is useless, as getSSLParameters() is always non-null.
Which Java version are you using? I'm suspicious that it may matter.
It seems weird to me that
foo.baris getting checked from the JDK'scheckServerTrusted(). The gRPC code is at least trying to disable that check. Thenullcheck is useless, as getSSLParameters() is always non-null.
zulu21.30.15-ca-jdk21.0.1-linux_x64
I just ran our tests with azul/zulu-openjdk:21.
openjdk 21.0.1 2023-10-17 LTS
OpenJDK Runtime Environment Zulu21.30+15-CA (build 21.0.1+12-LTS)
OpenJDK 64-Bit Server VM Zulu21.30+15-CA (build 21.0.1+12-LTS, mixed mode, sharing)
The tests passed, so I commented out the setEndpointIdentificationAlgorithm(null) in XdsX509TrustManager. That caused the tests to fail:
io.grpc.xds.internal.security.trust.XdsX509TrustManagerTest > checkServerTrustedSslEngine FAILED
java.security.cert.CertificateException: No subject alternative DNS name matching peer-host-from-mock found.
at java.base/sun.security.util.HostnameChecker.matchDNS(HostnameChecker.java:207)
at java.base/sun.security.util.HostnameChecker.match(HostnameChecker.java:103)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:457)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:431)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:291)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144)
at io.grpc.xds.internal.security.trust.XdsX509TrustManager.checkServerTrusted(XdsX509TrustManager.java:244)
at io.grpc.xds.internal.security.trust.XdsX509TrustManagerTest.checkServerTrustedSslEngine(XdsX509TrustManagerTest.java:535)
The failure path is close but not identical, at least partly because the test certs had a different SAN setup. But I think the important part is X509TrustManagerImpl.java:291. That line should only be executed if identityAlg != null. I'm at a loss, but it does make me wonder if something caused getSSLParameters() to return null.
stepping through I see that after disabling the check in sslParams, setSSLParameters in SSLConfiguration is ignoring it : https://github.com/openjdk/jdk/blob/8e12053e0352a26ecd7f2b9bc298ddb8fb4bb61b/src/java.base/share/classes/sun/security/ssl/SSLConfiguration.java#L312
I thought I put a comment here, but now I wonder if I put it somewhere else... I noticed AdvancedTlsX509TrustManager sets the endpoint identity algorithm to "", not null.
For posterity, both netty-tcnative and Conscrypt appear to observe null in their SSLEngines. So this appears to have only impacted SunJSSE.
XdsX509TrustManagerTest failing or passing doesn't tell us much, because it uses a fake SSLEngine.