HttpClientZioBackend fails with 'no statuscode in response'

[nickelsen]

While upgrading from AsyncHttpClientZioBackend to HttpClientZioBackend, we encountered this unexpected behavior.

I've created an example here, where AsyncHttpClientZioBackend works but HttpClientZioBackend fails for the same request: https://github.com/nickelsen/sttp-example

[Pask423]

It seems to be a problem of HTTP/2 support in HttpClient itself. As workaround changing version to HTTP/1.1 seems to fix the problem - below as quick way on how to do it. Next week I will try to add support for HTTP version changing in more reasonable way.

val changeVersion = (v: HttpRequest) => HttpRequest
  .newBuilder(v, (n: String, c: String) => true)
  .version(HttpClient.Version.HTTP_1_1)
  .build()

val httpClientZioBackend = HttpClientZioBackend(customizeRequest = changeVersion)
  .flatMap(s => s.send(request))

[nickelsen]

Thanks a lot for the workaround!

[nickelsen]

After moving to HttpClientZioBackend we get a ReadException due to HTTP/1.1 header parser received no bytes frequently from a couple of endpoints. Not sure if it's related to HTTP/1.1 or something unrelated in the underlying HttpClient.

[nickelsen]

...which is actually caused by an underlying java.net.SocketException: Connection reset.

[Pask423]

Could you give me some more details - stack trace, example, endpoint uri ?

[nickelsen]

The endpoints we see it on currently are authenticated, so I cannot give working example.

I'm also not able to reproduce in an isolated example.

We explored if this was related to our backend not being scoped (by accident), but even after we scoped it, the resets still occur.

We see the resets in about 33% of calls at the moment, but only on 1 out of the 14 endpoints we call with sttp.

We get these headers in the response:

< HTTP/1.1 200 OK
< server: envoy
< date: Wed, 09 Nov 2022 14:11:40 GMT
< content-type: application/json
< content-length: 88
< x-envoy-upstream-service-time: 31
< strict-transport-security: max-age=31536000
< Via: 1.1 google
< Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000

The stack trace looks like this:

zio.FiberFailure: sttp.client3.SttpClientException$ReadException: Exception when sending request: POST https://gatewayapi.com/rest/mtsms?token=<token>
	at sttp.client3.impl.zio.RIOMonadAsyncError.error(RIOMonadAsyncError.scala:24)
	at sttp.client3.impl.zio.RIOMonadAsyncError.flatMap(RIOMonadAsyncError.scala:12)
	[...our code...]
Caused by: sttp.client3.SttpClientException$ReadException: Exception when sending request: POST https://gatewayapi.com/rest/mtsms?token=<token>
	at sttp.client3.SttpClientException$.defaultExceptionToSttpClientException(SttpClientException.scala:46)
	at sttp.client3.HttpClientAsyncBackend.$anonfun$adjustExceptions$1(HttpClientAsyncBackend.scala:140)
	at sttp.client3.SttpClientException$$anonfun$adjustExceptions$1.applyOrElse(SttpClientException.scala:59)
	at sttp.client3.SttpClientException$$anonfun$adjustExceptions$1.applyOrElse(SttpClientException.scala:58)
	at zio.ZIO.$anonfun$catchSome$1(ZIO.scala:356)
	at scala.util.Either.fold(Either.scala:190)
	at zio.ZIO.tryRescue$1(ZIO.scala:356)
	at zio.ZIO.$anonfun$catchSome$4(ZIO.scala:358)
	at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:1149)
	at zio.internal.FiberRuntime.evaluateEffect(FiberRuntime.scala:384)
	at zio.internal.FiberRuntime.evaluateMessageWhileSuspended(FiberRuntime.scala:496)
	at zio.internal.FiberRuntime.drainQueueOnCurrentThread(FiberRuntime.scala:224)
	at zio.internal.FiberRuntime.run(FiberRuntime.scala:137)
	at zio.internal.ZScheduler$$anon$3.run(ZScheduler.scala:457)
Caused by: java.io.IOException: HTTP/1.1 header parser received no bytes
	at java.net.http/jdk.internal.net.http.common.Utils.wrapWithExtraDetail(Utils.java:348)
	at java.net.http/jdk.internal.net.http.Http1Response$HeadersReader.onReadError(Http1Response.java:675)
	at java.net.http/jdk.internal.net.http.Http1AsyncReceiver.checkForErrors(Http1AsyncReceiver.java:302)
	at java.net.http/jdk.internal.net.http.Http1AsyncReceiver.flush(Http1AsyncReceiver.java:268)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$LockingRestartableTask.run(SequentialScheduler.java:205)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run(SequentialScheduler.java:149)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:230)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.net.SocketException: Connection reset
	at java.base/sun.nio.ch.SocketChannelImpl.throwConnectionReset(SocketChannelImpl.java:394)
	at java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:426)
	at java.net.http/jdk.internal.net.http.SocketTube.readAvailable(SocketTube.java:1170)
	at java.net.http/jdk.internal.net.http.SocketTube$InternalReadPublisher$InternalReadSubscription.read(SocketTube.java:833)
	at java.net.http/jdk.internal.net.http.SocketTube$SocketFlowTask.run(SocketTube.java:181)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:230)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:303)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:256)
	at java.net.http/jdk.internal.net.http.SocketTube$InternalReadPublisher$InternalReadSubscription.signalReadable(SocketTube.java:774)
	at java.net.http/jdk.internal.net.http.SocketTube$InternalReadPublisher$ReadEvent.signalEvent(SocketTube.java:957)
	at java.net.http/jdk.internal.net.http.SocketTube$SocketFlowEvent.handle(SocketTube.java:253)
	at java.net.http/jdk.internal.net.http.HttpClientImpl$SelectorManager.handleEvent(HttpClientImpl.java:979)
	at java.net.http/jdk.internal.net.http.HttpClientImpl$SelectorManager.lambda$run$3(HttpClientImpl.java:934)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at java.net.http/jdk.internal.net.http.HttpClientImpl$SelectorManager.run(HttpClientImpl.java:934)

[Pask423]

After some checks on our side I think that is most likely a problem with HttpClient itself or something with underlying Http connection. Unfortunately I was unable to reproduce it ether with sttp or pure HttpClient.

[nickelsen]

Thanks a lot for looking into it! 🤩

I think we're just going to handle the resets (or whatever is going on) on our end, also given that the requests actually complete. I think that report can be ignored from the original issue.

Besides that, the upgrade from AsyncHttpClientZioBackend to HttpClientZioBackend is running smoothly. 🙌

[Pask423]

FYI Since v3.8.5 there is a cleaner way to apply the workaround from above Just:

val requestWithVersion = request.httpVersion(HTTP_1_1)
val httpClientZioBackend = HttpClientZioBackend()
  .flatMap(s => s.send(requestWithVersion))

[ipetkovic]

@nickelsen did you perhaps manage to workaround the issue? @Pask423 any update on this one?

I have the same problem using HttpClientZioBackend. I have been using sttp client AsyncHttpClientZioBackend without issues and with upgrade to sttp client 3.8.15 I get these:

sttp.client3.SttpClientException$ReadException: Exception when sending request: POST <url> : java.net.SocketException: Connection reset.

I tried to use http 1.1 workaround but it didn't help. I cannot reproduce it locally but this happens in production app environment with many requests being sent. This happens occasionally, not always. In 1% of requests or less.

[adamw]

@ipetkovic this might be a separate issue (the error message is different), but maybe you can try updating Java, or checking your Java version? That's where the core implementation of the http client lives.

[ipetkovic]

@adamw yeah, you are probably right. I could have created separate issue. Was writing here since in one of the comments same error was mentioned. I suspect that it is related to HttpClient and not HttpClientZioBackend. But did not test it. I just went back to AsyncHttpClientZioBackend and it solved my problem so I stopped looking into it. I did not realize before that it is still available in the new version. I am using java 17.0.2.

[adamw]

@ipetkovic Indeed, async-http-backend was deprecated for some time, but now it's back since a new maintainer stepped in.