[xDS] Client side Istio proxy fails to connect if gRPC server does not use netty-tcnative

[cfredri4]

Full disclosure: this issue was not found using grpc-xds, but I raise it as a potential issue for grpc-xds because I suspect it can happen there also.

Here's the setup:

I'm doing a "lite" version of proxyless gRPC servers in Istio without using xDS. On client side I configure a normal plaintext channel, which is intercepted by client-side Istio proxy and sent over TLS to server. On server side I bypass the server-side Istio proxy and instead configure the server like a normal TLS server but using Istio proxy's certificates. I.e. like this: Plaintext client channel -> client-side Istio proxy (does TLS) -> server with TLS using Istio proxy's certificates This works great, as long as netty-tcnative is used. If I don't have netty-tcnative (i.e. use OpenJDK TLS) then I get the below error on server (on debug) and client-side istio proxy fails to connect. It looks like the client-side Istio proxy sends something "interesting" (and invalid) as SNI.

This isn't really an issue for my use case as I always have netty-tcnative (or shaded netty), but I wanted to raise it as I suspect it can happen with grpc-xds also under the same circumstances (i.e. server using grpc-xds without netty-tcnative). Feel free to close if not relevant. 😅

I've been unable to track down where specifically a fix (if needed) could be done, but I can attempt a PR if someone has any ideas or is able to point me in the right direction.

Caused by: io.netty.handler.codec.DecoderException: javax.net.ssl.SSLProtocolException: Illegal server name, type=host_name(0), name=outbound_.9090_._.my-app.my-namespace.svc.cluster.local, value={6F7574626F756E645F2E393039305F2E5F2E6D792D6170702E6D792D6E616D6573706163652E7376632E636C75737465722E6C6F63616C}
at io.netty.handler.ssl.SslHandler$SslTasksRunner.wrapIfNeeded(SslHandler.java:1756)
... 13 common frames omitted
Caused by: javax.net.ssl.SSLProtocolException: Illegal server name, type=host_name(0), name=outbound_.9090_._.my-app.my-namespace.svc.cluster.local, value={6F7574626F756E645F2E393039305F2E5F2E6D792D6170702E6D792D6E616D6573706163652E7376632E636C75737465722E6C6F63616C}
at java.base/sun.security.ssl.ServerNameExtension$CHServerNamesSpec.<init>(Unknown Source)
at java.base/sun.security.ssl.ServerNameExtension$CHServerNameConsumer.consume(Unknown Source)
at java.base/sun.security.ssl.SSLExtension.consumeOnLoad(Unknown Source)
at java.base/sun.security.ssl.SSLExtensions.consumeOnLoad(Unknown Source)
at java.base/sun.security.ssl.ClientHello$T13ClientHelloConsumer.consume(Unknown Source)
at java.base/sun.security.ssl.ClientHello$ClientHelloConsumer.onClientHello(Unknown Source)
at java.base/sun.security.ssl.ClientHello$ClientHelloConsumer.consume(Unknown Source)
at java.base/sun.security.ssl.SSLHandshake.consume(Unknown Source)
at java.base/sun.security.ssl.HandshakeContext.dispatch(Unknown Source)
at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(Unknown Source)
at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(Unknown Source)
at java.base/java.security.AccessController.doPrivileged(Unknown Source)
at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(Unknown Source)
at io.netty.handler.ssl.SslHandler$SslTasksRunner.run(SslHandler.java:1887)
at java.base/java.util.concurrent.ThreadPerTaskExecutor$TaskRunner.run(Unknown Source)
at java.base/java.lang.VirtualThread.run(Unknown Source)
Caused by: java.lang.IllegalArgumentException: The encoded server name value is invalid
at java.base/javax.net.ssl.SNIHostName.<init>(Unknown Source)
... 16 common frames omitted
Caused by: java.lang.IllegalArgumentException: Contains non-LDH ASCII characters
at java.base/java.net.IDN.toASCIIInternal(Unknown Source)
at java.base/java.net.IDN.toASCII(Unknown Source)
... 17 common frames omitted