io.grpc.StatusRuntimeException: INTERNAL: Encountered end-of-stream mid-frame when calling grpc service using client code

[Selvapandian-b]

What version of gRPC-Java are you using?

1.42.1

What is your environment?

windows, jdk - 11, grpc-spring-boot - 2.13.0.RELEASE

What did you expect to see?

Retrieving all the records (findAll) Total records - 10100. It throws error at 8200th record

What did you see instead?

Its fails at around 8200th record. Not all the time. When we make 4 calls out of that 1 call fails and return the error

Steps to reproduce the bug

When keep on hitting a service multiple times (around 4 times) one call throws error randomly.

Error log

Exception in thread "main" io.grpc.StatusRuntimeException: INTERNAL: Encountered end-of-stream mid-frame
	at io.grpc.Status.asRuntimeException(Status.java:533)
	at io.grpc.stub.ClientCalls$BlockingResponseStream.hasNext(ClientCalls.java:648)

[temawi]

If I had to guess there might be something wrong with how you manage connection states, but without seeing the code I can't be sure.

[temawi]

Some follow-up questions to better understand if thsi might be a bug in gRPC:

• What version do you have on the server?
• Is it also a Java server?
• Are there any proxies involved?
• Do you have a deadline set on the RPC?

[Selvapandian-b]

Hi temawi,

Below are the details required.

1.version on the server - grpc java - 1.42.1 2.grpc-spring-boot 2.13.0-release is deployed in gcp (Google cloud kubernetes server) 3.Load balancer is configured in gcp. 4.We didn't configure any deadline.

We are getting this error for normal less time taking services as well. Not all the time we are getting this error. In a ratio of one among five calls throws the "Encountered end-of-stream mid-frame" error. On server there is no error log instead its printing the call completed log. We are using grpc java client using blocking stub to make calls to the server.

[temawi]

Sorry for the delay, some more questions:

• Do you also use version 1.42.1 on the client?
• Have you tried with the latest version of grcp-java?
• Are you able to run a test with a single call while having http2 frame level logging enabled? We could try to debug further with that and maybe determine if this is something in grpc-java or if maybe the load balancer is misbehaving.

You can enable this logging by providing a this file as a value to java.util.logging.config.file system property.

handlers = java.util.logging.ConsoleHandler
java.util.logging.ConsoleHandler.level = ALL

io.grpc.netty.NettyClientHandler.level = FINE
io.grpc.netty.NettyServerHandler.level = FINE
io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler.level = FINE
io.grpc.netty.shaded.io.grpc.netty.NettyServerHandler.level = FINE

[ejona86]

No response to provide more information, so closing. More information can be provided and then we can reopen.