H2 Clear-Text protocol error on upgrade to spring-cloud 2021.0.1

[mateofacu]

Describe the bug We have our api gateway configured to use http2 (http2: server.http2.enabled= true)

• spring-cloud: 2021.0.0

After upgrading to spring-cloud 2021.0.1 we get a netty protocol exception. Downgrading again to 2021.0.0 solves the problem.

Stacktrace: fullstacktrace.txt

java.lang.IllegalArgumentException: Configured H2 protocol without TLS. Use H2 Clear-Text protocol via HttpClient#protocol or configure TLS via HttpClient#secure at reactor.netty.http.client.HttpClientConnect$MonoHttpConnect.lambda$subscribe$0(HttpClientConnect.java:251) ~[reactor-netty-http-1.0.15.jar:1.0.15] Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: Error has been observed at the following site(s): *__checkpoint ⇢ org.springframework.cloud.gateway.filter.WeightCalculatorWebFilter [DefaultWebFilterChain] *__checkpoint ⇢ org.springframework.security.web.server.authorization.AuthorizationWebFilter [DefaultWebFilterChain] *__checkpoint ⇢ org.springframework.security.web.server.authorization.ExceptionTranslationWebFilter [DefaultWebFilterChain] *__checkpoint ⇢ org.springframework.security.web.server.authe ntication.logout.LogoutWebFilter [DefaultWebFilterChain] *__checkpoint ⇢ org.springframework.security.web.server.savedrequest.ServerRequestCacheWebFilter [DefaultWebFilterChain] *__checkpoint ⇢ org.springframework.security.web.server.context.SecurityContextServerWebExchangeWebFilter [DefaultWebFilterChain] *__checkpoint ⇢ org.springframework.security.web.server.context.ReactorContextWebFilter [DefaultWebFilterChain] *__checkpoint ⇢ org.springframework.security.web.server.header.HttpHeaderWriterWebFilter [DefaultWebFilterChain] *__checkpoint ⇢ org.springframework.security.config.web.server.ServerHttpSecurity$ServerWebExchangeReactorContextWebFilter [DefaultWebFilterChain] *__checkpoint ⇢ org.springframework.security.web.server.WebFilterChainProxy [DefaultWebFilterChain] *__checkpoint ⇢ org.springframework.boot.actuate.metrics.web.reactive.server.MetricsWebFilter [DefaultWebFilterChain] *__checkpoint ⇢ HTTP POST "/api/xxxxx" [ExceptionHandlingWebHandler]

[DennieBroucke]

Hi @mateofacu ,

The problem lies in the HttpClientFactory; the create instance method contains the following code if (serverProperties.getHttp2().isEnabled()) { httpClient = httpClient.protocol(HttpProtocol.HTTP11, HttpProtocol.H2); } However, this causes problems when dealing with http connections or a mix of http and https connections. A simple workaround could be to make a HttpClientCustomizer that sets the protocols to a more sensible default httpClient = httpClient.protocol(HttpProtocol.HTTP11, HttpProtocol.H2, HttpProtocol.H2C) or the specific protocols for your connections

[mateofacu]

Hi @DennieBroucke ,

I didn't know about HttpClientCustomizer interface. I gave it a try but unfortunately we are still getting the same error. I checked that the httpClient was correctly configured using HttpProtocol.HTTP11, HttpProtocol.H2, HttpProtocol.H2C.

The internal routes are "http". The services are using h2 enabled feature. I try the same test using http 1.1 but still I get the same error.

I added a full stack trace.

Any ideas are welcome.

[mateofacu]

I verified that spring-cloud-gateway-server 3.1.0 code has the same default protocol configuration than 3.1.1.

In NettyConfiguration

if (serverProperties.getHttp2().isEnabled()) { httpClient = httpClient.protocol(HttpProtocol.HTTP11, HttpProtocol.H2); }

[DennieBroucke]

Apologies for giving you false hope, we had some success, but after checking I've noticed we then had some other issues and reverted to a previous version of spring cloud as well.

[mateofacu]

Apologies for giving you false hope, we had some success, but after checking I've noticed we then had some other issues and reverted to a previous version of spring cloud as well.

Any help is welcome. I realized HttpClient creation has been refactored in version 3.1.1. May be there is a configuration difference between both versions.

[ypasmk]

Is there any update for this? It seems like a blocker to me ... Is there any workaround or plan? It seems that after spring cloud gateway version 3.1.0, having HTTP as downstream services is not supported and 3.1.0 is listed with 1 vulnerability.

[hingbong]

got a same error too.

[cscheffel]

got this with spring-cloud: 2021.0.3 too

[amrynsky]

looks like the breaking change. getting the same after upgrading to 2021.0.1.

as a workaround, we disabled http2 support for the server by setting --server.http2.enabled=false

[amrynsky]

Here are details for this issue:

• by default spring cloud gateway enables 2 protocols HTTP11 & H2
• H2 is not supported for HTTP protocol
• underlining Reactor Netty is trying to remove H2 for configuration without SSL Context, but only in case sslProvider != null https://github.com/reactor/reactor-netty/blob/main/reactor-netty-http/src/main/java/reactor/netty/http/client/HttpClientConnect.java#L247
• the previous version used "noop" SSL provider for H2 https://github.com/spring-cloud/spring-cloud-gateway/blob/9d178e2217d2f0ca12221dea8a2bdea844e5ac27/spring-cloud-gateway-server/src/main/java/org/springframework/cloud/gateway/config/GatewayAutoConfiguration.java#L720
• In the latest version this logic was removed and as a result Reactor Netty could not remove H2 and throws the exception https://github.com/reactor/reactor-netty/blob/main/reactor-netty-http/src/main/java/reactor/netty/http/client/HttpClientConnect.java#L253

In case you want to use spring cloud gateway to route to http, H2 should be disabled explicitly using server.http2.enabled=false because it will not be used anyway.

[neeravsv30]

Are we planning to fix this? Even we are getting the same issue with recent Spring cloud version. We want to use http2 so we don’t have an option to disable it

[mjesseni]

Hi There! First of all, I would like to thank you all for your great work. I think, I could work around this issue by providing a custom HttpClientSslConfigurer bean replacing the one from the GatewayAutoConfiguration by marking the bean primary. In the #configureSsl method I did the following which seems to work:

 @Bean
  @Primary
  public HttpClientSslConfigurer noopHttpClientSslConfigurer(HttpClientProperties httpClientProperties,
      final ServerProperties serverProperties) {
    return new HttpClientSslConfigurer(httpClientProperties.getSsl(), serverProperties) {
      @Override
      public HttpClient configureSsl(HttpClient client) {
        if(serverProperties.getHttp2().isEnabled()) {
          HttpClientProperties.Ssl ssl = httpClientProperties.getSsl();
          return client.secure(sslContextSpec -> {
            Http2SslContextSpec clientSslCtxt = Http2SslContextSpec.forClient()
                .configure(builder -> builder.trustManager(InsecureTrustManagerFactory.INSTANCE));
            sslContextSpec.sslContext(clientSslCtxt).handshakeTimeout(ssl.getHandshakeTimeout())
                .closeNotifyFlushTimeout(ssl.getCloseNotifyFlushTimeout())
                .closeNotifyReadTimeout(ssl.getCloseNotifyReadTimeout());
          });
        }
        return super.configureSsl(client);
      }
    };
  }

By doing this, we can still use HTTPS on the gateway and HTTP communication behind the gateway even if HTTP2 is active.

Many wishes, Markus

[spencergibb]

Sorry for the late reply. So, given @mjesseni's workaround, if we add an option to enable h2c, we could setup the http2 ssl context to use a InsecureTrustManagerFactory?