VisualVM hangs with 100% cpu usage

[cowwoc]

Describe the bug If I leave VisualVM running for a couple of hours in the Monitor tab then eventually its window stops repainting properly. I see it using 100% cpu in spite of not being able to interact with it at all.

This issue could be related to https://github.com/oracle/visualvm/issues/101 though I suspect you have upgraded the Marlin rendering engine that ships with VisualVM at this point.

VisualVM log Please attach VisualVM messages.log file to help diagnose your problem.

messages.log is available via Help | About | Logfile and via VisualVM | About VisualVM | Logfile on macOS

Screenshots If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: Microsoft Windows [Version 10.0.19044.2006]
• JDK version: zulu8.64.0.19-ca-jdk8.0.345-win_x64
• Version: 2.1.5

Additional context messages.log visualvm-stacktrace.txt

Looking at messages.log, I can see VisualVM is running out of memory but it's not clear to me why. I just opened it and attached it to monitor two JVMs. This typical usage should not cause this kind of crash.

[thurka]

The splitted heap dump seems to be messed up. I am not able to combine zip file from it. Anyway visualvm is running with -XX:+HeapDumpOnOutOfMemoryError so in case of OutOfMemoryError the correct heap dump is in C:\Users\Gili\AppData\Roaming\VisualVM\2.1.5\var\log\ can you please upload this heap dump somewhere (google drive, etc.) and add link here. Thanks.

[thurka]

This issue could be related to #101 though I suspect you have upgraded the Marlin rendering engine that ships with VisualVM at this point.

This is not related to #101, since you are running on JDK 8. FYI: Marlin rendering engine is part of JDK, not VisualVM.

[cowwoc]

Please see https://youtrack.jetbrains.com/issue/IDEA-305072. Maybe this is one and the same (JDK) bug? This is the application I am attaching VisualVM to.

[thurka]

I don't see how this is related to this bug. Can you please provide the heap dump I asked for? This could give me something I can investigate.

[cowwoc]

@thurka I've posted an updated heap dump for your review at [removed]. Please let me know when you are done downloading it so I can remove the file. Thank you.

[thurka]

Thanks, I downloaded the heap dump.

[cowwoc]

@thurka I again think that this issue is related to the JDK hang bug I referenced earlier. If you take a look at https://bugs.openjdk.org/browse/JDK-8296463 Fairoz Matte is only able to reproduce the JDK hang if VisualVM is attached. Something screwy is going on.

Maybe once you fix this bug, the other bug will go away as well. Did you have any luck with the heap dump?

[w6et]

try change
-XX:HeapDumpPath=C:\Users\Gili\AppData\Roaming\VisualVM\2.1.5\var\log\heapdump.hprof
to 
-XX:HeapDumpPath=C:\Users\Gili\AppData\Roaming\VisualVM\2.1.5\var\log\

I'm not sure,I guess "monitor two JVMs",maybe operate same file or something else, occured an exception:" Could not open PerfMemory", then cause rmi.newSocket again again(maybe some required arguments is missing)--->SEVERE [null]: Last record repeated again.-->OOM

[thurka]

Did you have any luck with the heap dump?

The heap dump shows that the monitored application creates enormenous number of new threads. Threads view needs to store some info for every thread, so this ultimately leads to OOME. The solution is to run VisualVM with increased Xmx. Default is 768M.

[cowwoc]

@thurka Okay, this is beginning to make more sense. Does VisualVM retain memory for dead threads? If so, increasing the maximum amount of memory won't help because ultimately it'll still run out of memory. As you correctly pointed out, we are creating (and destroying) millions of threads per hour. With JDK 19's ThreadPerTask executor this is going to become a common use-case.

[thurka]

Does VisualVM retain memory for dead threads?

Yes, you can display data for dead threads, so the info for already dead threads are preserved.

If so, increasing the maximum amount of memory won't help because ultimately it'll still run out of memory.

If you plan to monitor it for very long time, yes. With increased Xmx, it can extend the monitoring time long enough, so you can find the information you are looking for (whatever it is).

As you correctly pointed out, we are creating (and destroying) millions of threads per hour. With JDK 19's ThreadPerTask executor this is going to become a common use-case.

Unless I misunderstood JEP 425: Virtual Threads, you should use virtual threads in this situation. You can try it for yourself. In PlatformThreadsTestcase just replace:

    private final ExecutorService blockingTasksExecutor = Executors.newThreadPerTaskExecutor(
            blockingTasksThreadFactory);

with

    private final ExecutorService blockingTasksExecutor = Executors.newVirtualThreadPerTaskExecutor();

[cowwoc]

Yes, you can display data for dead threads, so the info for already dead threads are preserved.

This will have to change. It doesn't matter how much memory we throw at VisualVM, the minute an application uses Executors.newThreadPerTaskExecutor VisualVM will run out of memory very quickly. This is increasingly true if anyone uses virtual threads.

Two options that come to mind:

1. Provide a way to disable tracking of Finished threads. If the monitored application uses virtual threads, warn users that they need to disable this feature to avoid OOME.
2. Add a configuration option that reads "Retain history for the last [X] finished threads".

I personally prefer option 2. It would work regardless of the Thread or Executor type being used. It's simple to understand and easy to configure.

Unless I misunderstood JEP 425: Virtual Threads, you should use virtual threads in this situation. You can try it for yourself. In PlatformThreadsTestcase just replace:

I think you misunderstood my point. I meant that once VisualVM adds support for virtual threads (i.e. the Threads tab shows virtual threads, not just carrier threads) then anyone using virtual threads will encounter this OOME problem fairly quickly. We need to fix this problem.

[cowwoc]

The JDK bug (https://bugs.openjdk.org/browse/JDK-8296463) and IDE bug (https://youtrack.jetbrains.com/issue/IDEA-305777) have both been fixed. The only remaining bug associated with the use of Executors.newThreadPerTaskExecutor is VisualVM.

Have you given any thought on how to solve this? I mentioned two options above. Maybe you have another approach in mind?

[thurka]

I think you misunderstood my point.

Definitely, you did not mention that you are trying to simulate virtual threads with ordinary ones.

I meant that once VisualVM adds support for virtual threads

I agree that current threads view cannot be used for the virtual threads. This needs to be resolved when VisualVM adds support for virtual threads, but we are not there yet. Virtual threads support is still an experimental feature and JDK does not provide much info about them. Current threads view is not suitable for virtual threads, since it displays not very useful information in a large area (the thread was finished). The virtual threads should be condesed somehow. I would like to line up them to row(s), so the maximum number of rows in the virtual threads view will be number of currently running (not death) virtual threads.

[cowwoc]

Sounds good. Let me know when you begin adding support for Virtual Threads and I'll happily take a look. Thanks in advance.

[cowwoc]

Dependent on https://github.com/oracle/visualvm/issues/462