grpc-java
grpc-java copied to clipboard
grpc
Blocking stub blocks indefinitely if client exceeds max outbound message size
What version of gRPC-Java are you using?
1.72.0
What is your environment?
Linux - Ubuntu 22.04.5 LTS JDK - jdk-corretto_21.0.1.12.1 Repro repository - https://github.com/jackrwoods/grpc-problem-repro
What did you expect to see?
In the console output, I expect to see "Sending message (Java)" followed by an exception/stacktrace that should interrupt and exit the application
What did you see instead?
In the console output, I see "Sending message (Java)" and an exception, but the application hangs and does not exit.
Steps to reproduce the bug
- Clone the repository https://github.com/jackrwoods/grpc-problem-repro
- Within the repository, run
./gradlew build - Within the repository, run
./gradlew run - Observe that the application does not exit.
When using a blocking stub to frame a grpc message, I am observing that my GRPC call parks the calling thread indefinitely. I always see a logged exception detailing why the request failed, but the GRPC call does not return.
I have set up a repro in this repository: jackrwoods/grpc-problem-repro. I've reproduced this behavior with a "blocking" stub (jackrwoods/grpc-problem-repro@main/src/main/java/org/example/Main.java) and a coroutine stub wrapped with a runBlocking (jackrwoods/grpc-problem-repro@main/src/main/kotlin/org/example/MainKotlinCoroutineVersion.kt). In both cases, the application hangs when the maximum GRPC message size is exceeded.
Is this user (me) error, or is there a workaround?
ClientCalls.blockingUnaryCall is stuck waiting for the responseFuture to be done. One time it gets a runnable given by ManagedChannelImpl that executes the call that fails with the exception thrown by MessageFramer about the outbound message being too large and the exception is swallowed. The main thread meanwhile parks itself again waiting for the response future to be completed and there are no more runnables ever added to the executor that could unpark it.
It looks like we should be putting a timeout waiting for the response future may be from CallOptions.deadline (and have the generated code have a way of accepting it), or take into account the number of unparks that would happen for a unary call.
Jul 02, 2025 8:48:39 PM io.grpc.stub.ClientCalls$ThreadlessExecutor runQuietly
WARNING: Runnable threw exception
io.grpc.StatusRuntimeException: INTERNAL: Failed to frame message
at io.grpc.Status.asRuntimeException(Status.java:524)
at io.grpc.internal.MessageFramer.writePayload(MessageFramer.java:153)
at io.grpc.internal.AbstractStream.writeMessage(AbstractStream.java:65)
at io.grpc.internal.ForwardingClientStream.writeMessage(ForwardingClientStream.java:37)
at io.grpc.internal.DelayedStream$6.run(DelayedStream.java:283)
at io.grpc.internal.DelayedStream.drainPendingCalls(DelayedStream.java:182)
at io.grpc.internal.DelayedStream.access$100(DelayedStream.java:44)
at io.grpc.internal.DelayedStream$4.run(DelayedStream.java:148)
at io.grpc.stub.ClientCalls$ThreadlessExecutor.runQuietly(ClientCalls.java:831)
at io.grpc.stub.ClientCalls$ThreadlessExecutor.waitAndDrain(ClientCalls.java:808)
at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:166)
at org.example.ExampleServiceGrpc$ExampleServiceBlockingStub.nothingDoer(ExampleServiceGrpc.java:157)
at org.example.Main.main(Main.java:19)
Caused by: io.grpc.StatusRuntimeException: RESOURCE_EXHAUSTED: message too large 13 > 0
at io.grpc.Status.asRuntimeException(Status.java:524)
at io.grpc.internal.MessageFramer.writeKnownLengthUncompressed(MessageFramer.java:219)
at io.grpc.internal.MessageFramer.writeUncompressed(MessageFramer.java:168)
at io.grpc.internal.MessageFramer.writePayload(MessageFramer.java:141)
... 11 more
Edit: definitely not grpc 1.72. It's actually a mix, with grpc-core and grpc-netty at version 1.46.
$ gradle dependencies --configuration runtimeClasspath
> Task :dependencies
------------------------------------------------------------
Root project 'blockingGrpcStub'
------------------------------------------------------------
runtimeClasspath - Runtime classpath of 'main'.
+--- com.google.protobuf:protobuf-kotlin -> 3.24.0
| +--- com.google.protobuf:protobuf-java:3.24.0 -> 3.25.5
| \--- org.jetbrains.kotlin:kotlin-stdlib:1.6.0 -> 2.0.20
| +--- org.jetbrains:annotations:13.0 -> 23.0.0
| +--- org.jetbrains.kotlin:kotlin-stdlib-common:2.0.20 (c)
| +--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.8.0 -> 1.8.20 (c)
| \--- org.jetbrains.kotlin:kotlin-stdlib-jdk7:1.8.0 -> 1.8.20 (c)
+--- org.jetbrains.kotlin:kotlin-stdlib:2.0.20 (*)
+--- io.grpc:grpc-netty:1.46.0
| +--- io.grpc:grpc-core:1.46.0
| | +--- io.grpc:grpc-api:1.46.0 -> 1.72.0
| | | +--- com.google.code.findbugs:jsr305:3.0.2
| | | +--- com.google.errorprone:error_prone_annotations:2.30.0
| | | \--- com.google.guava:guava:33.3.1-android
| | | +--- com.google.guava:failureaccess:1.0.2
| | | +--- com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
| | | +--- com.google.code.findbugs:jsr305:3.0.2
| | | +--- org.checkerframework:checker-qual:3.43.0
| | | +--- com.google.errorprone:error_prone_annotations:2.28.0 -> 2.30.0
| | | \--- com.google.j2objc:j2objc-annotations:3.0.0
| | +--- com.google.code.gson:gson:2.8.9
| | +--- com.google.android:annotations:4.1.1.4
| | +--- org.codehaus.mojo:animal-sniffer-annotations:1.19 -> 1.24
| | +--- com.google.errorprone:error_prone_annotations:2.10.0 -> 2.30.0
| | +--- com.google.guava:guava:31.0.1-android -> 33.3.1-android (*)
| | \--- io.perfmark:perfmark-api:0.25.0
| +--- io.netty:netty-codec-http2:4.1.72.Final
| | +--- io.netty:netty-common:4.1.72.Final
| | +--- io.netty:netty-buffer:4.1.72.Final
| | | \--- io.netty:netty-common:4.1.72.Final
| | +--- io.netty:netty-transport:4.1.72.Final
| | | +--- io.netty:netty-common:4.1.72.Final
| | | +--- io.netty:netty-buffer:4.1.72.Final (*)
| | | \--- io.netty:netty-resolver:4.1.72.Final
| | | \--- io.netty:netty-common:4.1.72.Final
| | +--- io.netty:netty-codec:4.1.72.Final
| | | +--- io.netty:netty-common:4.1.72.Final
| | | +--- io.netty:netty-buffer:4.1.72.Final (*)
| | | \--- io.netty:netty-transport:4.1.72.Final (*)
| | +--- io.netty:netty-handler:4.1.72.Final
| | | +--- io.netty:netty-common:4.1.72.Final
| | | +--- io.netty:netty-resolver:4.1.72.Final (*)
| | | +--- io.netty:netty-buffer:4.1.72.Final (*)
| | | +--- io.netty:netty-transport:4.1.72.Final (*)
| | | +--- io.netty:netty-codec:4.1.72.Final (*)
| | | \--- io.netty:netty-tcnative-classes:2.0.46.Final
| | \--- io.netty:netty-codec-http:4.1.72.Final
| | +--- io.netty:netty-common:4.1.72.Final
| | +--- io.netty:netty-buffer:4.1.72.Final (*)
| | +--- io.netty:netty-transport:4.1.72.Final (*)
| | +--- io.netty:netty-codec:4.1.72.Final (*)
| | \--- io.netty:netty-handler:4.1.72.Final (*)
| +--- io.netty:netty-handler-proxy:4.1.72.Final
| | +--- io.netty:netty-common:4.1.72.Final
| | +--- io.netty:netty-buffer:4.1.72.Final (*)
| | +--- io.netty:netty-transport:4.1.72.Final (*)
| | +--- io.netty:netty-codec:4.1.72.Final (*)
| | +--- io.netty:netty-codec-socks:4.1.72.Final
| | | +--- io.netty:netty-common:4.1.72.Final
| | | +--- io.netty:netty-buffer:4.1.72.Final (*)
| | | +--- io.netty:netty-transport:4.1.72.Final (*)
| | | \--- io.netty:netty-codec:4.1.72.Final (*)
| | \--- io.netty:netty-codec-http:4.1.72.Final (*)
| +--- com.google.guava:guava:31.0.1-android -> 33.3.1-android (*)
| +--- com.google.errorprone:error_prone_annotations:2.10.0 -> 2.30.0
| \--- io.perfmark:perfmark-api:0.25.0
+--- io.grpc:grpc-protobuf:1.72.0
| +--- io.grpc:grpc-api:1.72.0 (*)
| +--- com.google.code.findbugs:jsr305:3.0.2
| +--- com.google.protobuf:protobuf-java:3.25.5
| +--- com.google.api.grpc:proto-google-common-protos:2.51.0
| | \--- com.google.protobuf:protobuf-java:3.25.5
| +--- com.google.guava:guava:33.3.1-android (*)
| \--- io.grpc:grpc-protobuf-lite:1.72.0
| +--- io.grpc:grpc-api:1.72.0 (*)
| +--- com.google.code.findbugs:jsr305:3.0.2
| \--- com.google.guava:guava:33.3.1-android (*)
+--- io.grpc:grpc-kotlin-stub:1.4.1
| +--- org.jetbrains.kotlin:kotlin-stdlib:1.8.0 -> 2.0.20 (*)
| +--- org.jetbrains.kotlinx:kotlinx-coroutines-core-jvm:1.7.3
| | +--- org.jetbrains:annotations:23.0.0
| | +--- org.jetbrains.kotlinx:kotlinx-coroutines-bom:1.7.3
| | | +--- org.jetbrains.kotlinx:kotlinx-coroutines-core-jvm:1.7.3 (c)
| | | \--- org.jetbrains.kotlinx:kotlinx-coroutines-core:1.7.3 (c)
| | +--- org.jetbrains.kotlin:kotlin-stdlib-common:1.8.20 -> 2.0.20
| | | \--- org.jetbrains.kotlin:kotlin-stdlib:2.0.20 (*)
| | \--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.8.20
| | +--- org.jetbrains.kotlin:kotlin-stdlib:1.8.20 -> 2.0.20 (*)
| | \--- org.jetbrains.kotlin:kotlin-stdlib-jdk7:1.8.20
| | \--- org.jetbrains.kotlin:kotlin-stdlib:1.8.20 -> 2.0.20 (*)
| +--- io.grpc:grpc-stub:1.57.2 -> 1.72.0
| | +--- io.grpc:grpc-api:1.72.0 (*)
| | +--- org.codehaus.mojo:animal-sniffer-annotations:1.24
| | +--- com.google.guava:guava:33.3.1-android (*)
| | \--- com.google.errorprone:error_prone_annotations:2.30.0
| \--- javax.annotation:javax.annotation-api:1.3.2
+--- io.grpc:grpc-stub:1.72.0 (*)
+--- io.grpc:grpc-bom:1.63.2
+--- com.google.protobuf:protoc:3.24.0
+--- com.google.protobuf:protobuf-java:3.24.0 -> 3.25.5
+--- com.google.protobuf:protobuf-kotlin:3.24.0 (*)
\--- org.jetbrains.kotlinx:kotlinx-coroutines-core:1.6.4 -> 1.7.3
\--- org.jetbrains.kotlinx:kotlinx-coroutines-core-jvm:1.7.3 (*)
(c) - A dependency constraint, not a dependency. The dependency affected by the constraint occurs elsewhere in the tree.
(*) - Indicates repeated occurrences of a transitive dependency subtree. Gradle expands transitive dependency subtrees only once per project; repeat occurrences only display the root of the subtree, followed by this annotation.
With this patch, to use 1.72:
diff --git a/build.gradle.kts b/build.gradle.kts
index c2375c6..0e90024 100644
--- a/build.gradle.kts
+++ b/build.gradle.kts
@@ -20,11 +20,11 @@ repositories {
dependencies {
- implementation("io.grpc:grpc-netty:1.46.0")
+ implementation("io.grpc:grpc-netty:1.72.0")
implementation("io.grpc:grpc-protobuf:1.72.0")
implementation("io.grpc:grpc-kotlin-stub:1.4.1")
implementation("io.grpc:grpc-stub:1.72.0")
- implementation("io.grpc:grpc-bom:1.63.2")
+ implementation("io.grpc:grpc-bom:1.72.0")
implementation("com.google.protobuf:protoc:3.24.0")
implementation("com.google.protobuf:protobuf-java:3.24.0")
implementation("com.google.protobuf:protobuf-kotlin:3.24.0")
@@ -43,7 +43,7 @@ protobuf {
}
plugins {
create("grpc") {
- artifact = "io.grpc:protoc-gen-grpc-java:1.53.0"
+ artifact = "io.grpc:protoc-gen-grpc-java:1.73.0"
}
create("grpckt") {
artifact = "io.grpc:protoc-gen-grpc-kotlin:1.4.1:jdk8@jar"
@@ -84,4 +84,4 @@ plugins.withId("org.jetbrains.kotlin.jvm") {
dependencies {
"api"("com.google.protobuf:protobuf-kotlin")
}
-}
\ No newline at end of file
+}
Jul 02, 2025 8:51:20 PM io.grpc.stub.ClientCalls$ThreadlessExecutor runQuietly
WARNING: Runnable threw exception
io.grpc.StatusRuntimeException: RESOURCE_EXHAUSTED: message too large 13 > 0
at io.grpc.Status.asRuntimeException(Status.java:524)
at io.grpc.internal.MessageFramer.writeKnownLengthUncompressed(MessageFramer.java:223)
at io.grpc.internal.MessageFramer.writeUncompressed(MessageFramer.java:176)
at io.grpc.internal.MessageFramer.writePayload(MessageFramer.java:147)
at io.grpc.internal.AbstractStream.writeMessage(AbstractStream.java:70)
at io.grpc.internal.ForwardingClientStream.writeMessage(ForwardingClientStream.java:37)
at io.grpc.internal.DelayedStream$6.run(DelayedStream.java:282)
at io.grpc.internal.DelayedStream.drainPendingCalls(DelayedStream.java:182)
at io.grpc.internal.DelayedStream.access$100(DelayedStream.java:44)
at io.grpc.internal.DelayedStream$4.run(DelayedStream.java:148)
at io.grpc.stub.ClientCalls$ThreadlessExecutor.runQuietly(ClientCalls.java:831)
at io.grpc.stub.ClientCalls$ThreadlessExecutor.waitAndDrain(ClientCalls.java:808)
at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:166)
at org.example.ExampleServiceGrpc$ExampleServiceBlockingStub.nothingDoer(ExampleServiceGrpc.java:199)
at org.example.Main.main(Main.java:19)
I've come across this issue updating tests for our gRPC based client application.
We're using grpc-java transitively via apache-arrow flight. The current version of grpc-core and grpc-netty is 1.71.
[INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ influxdb3-java ---
[INFO] com.influxdb:influxdb3-java:jar:1.3.0-SNAPSHOT
[INFO] +- org.apache.arrow:flight-core:jar:18.3.0:compile
[INFO] | +- org.apache.arrow:arrow-format:jar:18.3.0:compile
[INFO] | | \- com.google.flatbuffers:flatbuffers-java:jar:25.2.10:compile
[INFO] | +- org.apache.arrow:arrow-vector:jar:18.3.0:compile
[INFO] | | +- com.fasterxml.jackson.core:jackson-core:jar:2.18.3:compile
[INFO] | | +- com.fasterxml.jackson.core:jackson-annotations:jar:2.18.3:compile
[INFO] | | +- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:jar:2.18.3:compile
[INFO] | | \- commons-codec:commons-codec:jar:1.18.0:compile
[INFO] | +- org.apache.arrow:arrow-memory-core:jar:18.3.0:compile
[INFO] | +- org.apache.arrow:arrow-memory-netty:jar:18.3.0:runtime
[INFO] | | \- org.apache.arrow:arrow-memory-netty-buffer-patch:jar:18.3.0:runtime
[INFO] | +- io.grpc:grpc-netty:jar:1.71.0:compile
[INFO] | | +- org.codehaus.mojo:animal-sniffer-annotations:jar:1.24:compile
[INFO] | | +- io.netty:netty-handler-proxy:jar:4.1.110.Final:runtime
[INFO] | | | \- io.netty:netty-codec-socks:jar:4.1.110.Final:runtime
[INFO] | | +- io.perfmark:perfmark-api:jar:0.27.0:runtime
[INFO] | | \- io.grpc:grpc-util:jar:1.71.0:runtime
[INFO] | +- io.grpc:grpc-core:jar:1.71.0:compile
[INFO] | | +- com.google.android:annotations:jar:4.1.1.4:runtime
[INFO] | | \- io.grpc:grpc-context:jar:1.71.0:runtime
[INFO] | +- io.grpc:grpc-protobuf:jar:1.71.0:compile
[INFO] | | +- com.google.api.grpc:proto-google-common-protos:jar:2.51.0:compile
[INFO] | | \- io.grpc:grpc-protobuf-lite:jar:1.71.0:runtime
[INFO] | +- io.grpc:grpc-stub:jar:1.71.0:compile
[INFO] | +- com.google.protobuf:protobuf-java-util:jar:4.30.2:compile
[INFO] | +- io.grpc:grpc-api:jar:1.71.0:compile
[INFO] | +- com.fasterxml.jackson.core:jackson-databind:jar:2.18.3:compile
[INFO] | \- javax.annotation:javax.annotation-api:jar:1.3.2:compile
[INFO] +- io.netty:netty-handler:jar:4.2.3.Final:compile
[INFO] | +- io.netty:netty-common:jar:4.2.3.Final:compile
[INFO] | +- io.netty:netty-resolver:jar:4.2.3.Final:compile
[INFO] | +- io.netty:netty-transport:jar:4.2.3.Final:compile
[INFO] | \- io.netty:netty-codec-base:jar:4.2.3.Final:compile
[INFO] +- io.netty:netty-buffer:jar:4.2.3.Final:compile
[INFO] +- io.netty:netty-codec:jar:4.2.3.Final:compile
[INFO] | +- io.netty:netty-codec-compression:jar:4.2.3.Final:compile
[INFO] | +- io.netty:netty-codec-protobuf:jar:4.2.3.Final:compile
[INFO] | \- io.netty:netty-codec-marshalling:jar:4.2.3.Final:compile
[INFO] +- io.netty:netty-codec-http2:jar:4.2.3.Final:compile
[INFO] | \- io.netty:netty-codec-http:jar:4.2.3.Final:compile
[INFO] +- io.netty:netty-transport-native-unix-common:jar:4.2.3.Final:compile
[INFO] +- io.netty:netty-tcnative-boringssl-static:jar:2.0.72.Final:compile
[INFO] | +- io.netty:netty-tcnative-classes:jar:2.0.72.Final:compile
[INFO] | +- io.netty:netty-tcnative-boringssl-static:jar:linux-x86_64:2.0.72.Final:compile
[INFO] | +- io.netty:netty-tcnative-boringssl-static:jar:linux-aarch_64:2.0.72.Final:compile
[INFO] | +- io.netty:netty-tcnative-boringssl-static:jar:osx-x86_64:2.0.72.Final:compile
[INFO] | +- io.netty:netty-tcnative-boringssl-static:jar:osx-aarch_64:2.0.72.Final:compile
[INFO] | \- io.netty:netty-tcnative-boringssl-static:jar:windows-x86_64:2.0.72.Final:compile
[INFO] +- com.google.protobuf:protobuf-java:jar:4.31.1:compile
[INFO] +- com.google.guava:guava:jar:33.4.0-jre:compile
[INFO] | +- com.google.guava:failureaccess:jar:1.0.2:compile
[INFO] | +- com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava:compile
[INFO] | +- com.google.code.findbugs:jsr305:jar:3.0.2:compile
[INFO] | \- org.checkerframework:checker-qual:jar:3.43.0:compile
[INFO] +- com.google.j2objc:j2objc-annotations:jar:3.0.0:compile
[INFO] +- com.google.code.gson:gson:jar:2.13.1:compile
[INFO] +- com.google.errorprone:error_prone_annotations:jar:2.36.0:compile
[INFO] +- org.slf4j:slf4j-api:jar:2.0.17:compile
[INFO] +- org.junit.platform:junit-platform-engine:jar:1.13.4:compile
[INFO] | +- org.opentest4j:opentest4j:jar:1.3.0:compile
[INFO] | +- org.junit.platform:junit-platform-commons:jar:1.13.4:compile
[INFO] | \- org.apiguardian:apiguardian-api:jar:1.1.2:compile
[INFO] +- org.junit.platform:junit-platform-launcher:jar:1.13.4:compile
[INFO] +- org.junit.jupiter:junit-jupiter-engine:jar:5.13.4:test
[INFO] | \- org.junit.jupiter:junit-jupiter-api:jar:5.13.4:test
[INFO] +- org.assertj:assertj-core:jar:3.27.3:test
[INFO] | \- net.bytebuddy:byte-buddy:jar:1.15.11:test
[INFO] +- com.squareup.okhttp3:mockwebserver3-junit5:jar:5.1.0:test
[INFO] | +- com.squareup.okhttp3:okhttp-jvm:jar:5.1.0:test
[INFO] | | \- com.squareup.okio:okio-jvm:jar:3.15.0:test
[INFO] | +- com.squareup.okhttp3:mockwebserver3:jar:5.1.0:test
[INFO] | \- org.jetbrains.kotlin:kotlin-stdlib:jar:2.2.0:test
[INFO] | \- org.jetbrains:annotations:jar:13.0:test
[INFO] \- org.slf4j:slf4j-simple:jar:2.0.17:test
[INFO] ------------------------------------------
When testing the maxOutboundMessageSize property to a value low enough to guarantee an exception, the exception is indeed thrown (see below) but then the ManagedChannel hangs indefinitely.
Exception in thread "grpc-default-executor-0" io.grpc.StatusRuntimeException: RESOURCE_EXHAUSTED: message too large 125 > 10
at io.grpc.Status.asRuntimeException(Status.java:524)
at io.grpc.internal.MessageFramer.writeKnownLengthUncompressed(MessageFramer.java:223)
at io.grpc.internal.MessageFramer.writeUncompressed(MessageFramer.java:176)
at io.grpc.internal.MessageFramer.writePayload(MessageFramer.java:147)
at io.grpc.internal.AbstractStream.writeMessage(AbstractStream.java:70)
at io.grpc.internal.ForwardingClientStream.writeMessage(ForwardingClientStream.java:37)
at io.grpc.internal.DelayedStream$6.run(DelayedStream.java:282)
at io.grpc.internal.DelayedStream.drainPendingCalls(DelayedStream.java:182)
at io.grpc.internal.DelayedStream.access$100(DelayedStream.java:44)
at io.grpc.internal.DelayedStream$4.run(DelayedStream.java:148)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1623)
This test for this can be found here
I've observed the same behaviour with 1.73 and 1.76 (the latest release at the time of writing).
Are there any known workarounds or the only way is to remove withMaxOutboundMessageSize() on the client side and do message size validation manually (e.g. in interceptors or before the call)?
Yes, until it is fixed that is the only workaround when using the BlockingClientCall.