corretto
java.lang.ClassCircularityError: jdk/internal/misc/VirtualThreads thrown when using virtual threads under load
Describe the bug
- java.lang.ClassCircularityError: jdk/internal/misc/VirtualThreads when running Spring based microservice on Kubernetes.
- This exception happens when in spring boot we enable virtual threads
-
spring: threads: virtual: enabled: true - Exception happens when service is under high load (many incomming HTTP requests)
- Only remediation is to restart the POD
To Reproduce
No steps to reproduce. Happens from time to time, and only when high load is there. So i cannot provide exact steps or reproducible example. Found two tickets over internet which might be related:
- https://github.com/open-telemetry/opentelemetry-java-instrumentation/issues/10181
- https://bugs.java.com/bugdatabase/view_bug?bug_id=8322846 (Oracle JDK).
Expected behavior
No exceptions
Screenshots
N/A
Platform information
- Ubuntu 22.04.4 LTS
- Spring Boot 3.2.6
- Instrumentation with open telemetry java agent
- Instrumenation with app dynamics java agent
- JDK: openjdk 21.0.3 2024-04-16 LTSOpenJDK Runtime Environment Corretto-21.0.3.9.1 (build 21.0.3+9-LTS)OpenJDK 64-Bit Server VM Corretto-21.0.3.9.1 (build 21.0.3+9-LTS, mixed mode)
Additional context
Full stacktrace: fulls_stacktrace.txt
JDK-8322846 is already backported to 21.0.4 upstream, and would land in Corretto 21.0.4 release next week.
That said, java.lang.ClassCircularityError does not look like JDK-8322846. CCE implicates the class transformation code is some way. We have seen issues when agent tries to load the class that is currently under retransformation (JDK-8164165 shows the JVM throws CCE instead of more proper LinkageError in that case).
So I believe this is a bug in either OTel or AppDynamics java agent. Look around their bug-trackers for possibly related ClassCircularityError fixes.
I agree with @shipilev. This is most likely caused by incorrect usage of bytecode transformation in one of your instrumentation agents. Take a look at ByteBuddy issue #1666 for more details. That issue also references similar issues in the Data Dog dd-trace and the Glowroot library.
I've recently also opened JDK-8335619: Add an @apiNote to j.l.i.ClassFileTransformer to warn about recursive class loading and ClassCircularityErrors and submitted a PR for it.
The supporting class jdk.internal.misc.VirtualThreads has been removed in the loom repo, as has jdk.internal.event.VirtualThreadPinnedEvent. This removes class loading from LockSupport.park that can happen when the first usage is in a class file transformer, which may be what is happening here. I hope to bring some of these changes into main line in advance of the object monitors work.
