eclipse-mat
Add dump reliability check if MAT's loaded DTFJ doesn't match the dump's DTFJ
| --- | --- | | Bugzilla Link | 582736 | | Status | ASSIGNED | | Importance | P3 normal | | Reported | Dec 13, 2023 13:31 EDT | | Modified | Dec 18, 2023 13:53 EDT | | Reporter | Kevin Grigorenko |
Description
The DTFJ implementations for IBM Java and IBM Semeru Runtimes are different and usually incompatible, particularly for core dumps.
MAT ships with a built-in update site for IBM Java DTFJ at https://public.dhe.ibm.com/ibmdl/export/pub/software/websphere/runtimes/tools/dtfj/
As of this writing, that update site was last updated in 2020 and is not actively maintained.
IBM also publishes standalone update sites and packaged MAT builds for both IBM Java DTFJ and IBM Semeru Runtimes DTFJ at https://www.ibm.com/support/pages/eclipse-memory-analyzer-tool-dtfj-and-ibm-extensions
However, as of this writing, those update sites are not signed.
In addition, I have not found time to restart the work to add an option to MAT to specify the DTFJ implementation or pass through to the underlying JVM: https://bugs.eclipse.org/bugs/show_bug.cgi?id=567819
So that is the current and unfortunate state of things.
With that noted, as IBM Semeru Runtimes usage picks up, it is becoming a more common issue that users running with MAT + IBM Java DTFJ are trying to load dumps produced by IBM Semeru Runtimes. This produces confusing fatal errors such as the following:
java.lang.NoClassDefFoundError: com.ibm.j9ddr.vm29.pointer.generated.GC_ArrayObjectModelPointer
at com.ibm.j9ddr.vm29.j9.gc.GCArrayObjectModel.from(GCArrayObjectModel.java:52)
at com.ibm.j9ddr.vm29.j9.gc.GCObjectModel.
at com.ibm.j9ddr.vm29.j9.gc.GCObjectModel_V1.
at com.ibm.j9ddr.vm29.j9.gc.GCObjectModel.from(GCObjectModel.java:59)
at com.ibm.j9ddr.vm29.j9.ObjectModel.
at com.ibm.j9ddr.vm29.j9.gc.GCObjectHeapIteratorAddressOrderedList_V1.advanceScanPointer(GCObjectHeapIteratorAddressOrderedList_V1.java:200)
at com.ibm.j9ddr.vm29.j9.gc.GCObjectHeapIteratorAddressOrderedList_V1.next(GCObjectHeapIteratorAddressOrderedList_V1.java:283)
at com.ibm.j9ddr.vm29.j9.walkers.HeapWalker.walk(HeapWalker.java:118)
at com.ibm.j9ddr.vm29.view.dtfj.java.j9.HeapObjectIterator.hasNext(HeapObjectIterator.java:50)
at com.ibm.j9ddr.vm29.view.dtfj.java.DTFJJavaHeap$1.hasNext(DTFJJavaHeap.java:118)
at org.eclipse.mat.dtfj.DTFJIndexBuilder.fill(DTFJIndexBuilder.java:1236) [...]
MAT should have some detection of this and provide a better error.
By Kevin Grigorenko on Dec 14, 2023 08:48
https://git.eclipse.org/c/mat/org.eclipse.mat.git/commit/?id=a192f1be11e2b005001eca1cea26b7150b6cf2bb
https://git.eclipse.org/r/c/mat/org.eclipse.mat/+/205821
By Andrew Johnson on Dec 17, 2023 05:31
Minor warning from the build:
Should org.eclipse.mat.dtfj.DTFJIndexBuilder$DumpReliabilityResult be a static inner class?\
This class is an inner class, but does not use its embedded reference to the object which created it. This reference makes the instances of the class larger, and may keep the reference to the creator object alive longer than necessary. If possible, the class should be made static.
Probably doesn't matter much.
By Kevin Grigorenko on Dec 18, 2023 13:53
Good catch. Made the change and verified locally and all tests pass.
https://git.eclipse.org/c/mat/org.eclipse.mat.git/commit/?id=42b50511976f6072f4f66c7a7a42e39c14ad7d7e
https://git.eclipse.org/r/c/mat/org.eclipse.mat/+/205880
Any update on this one? Getting same error: The loaded DTFJ implementation for IBM Java does not match the file source of IBM Semeru Runtimes. Extracted Java version: JRE 21 Linux amd64-64 (build 21.0.5+11-LTS)
@pavelkokush The DTFJ update site at https://public.dhe.ibm.com/ibmdl/export/pub/software/websphere/runtimes/tools/dtfj/ is only supported for IBM Java (and it's also a very old build). If you want to parse IBM Semeru Runtimes dumps, then you need to use https://www.ibm.com/support/pages/eclipse-memory-analyzer-tool-dtfj-and-ibm-extensions
So, it's basically the same status though we're working on open sourcing the DTFJ plugin which will at some point hopefully allow integrating the Semeru plugin directly into MAT.
Hello, i tried to download MAT with dtfj extension and plugin from your link : https://www.ibm.com/support/pages/eclipse-memory-analyzer-tool-dtfj-and-ibm-extensions
but it seems that ibm doesn't allow to view this page without being a customer of IBM product. Do you know how to get this MAT with dtfj Semeru Runtimes?
@MansuyDavid These changes are being done IBM corporate wide so out of my control; however, I did just change it so that all you need to download is a free IBMid.
I do still have it on my plate to update https://public.dhe.ibm.com/ibmdl/export/pub/software/websphere/runtimes/tools/dtfj/ which was last updated in 2020 but that requires a lot of bureaucratic things that I haven't found the time for.
Thank you very much @kgibm for allowing free IBMid to download the MAT with DTFJ Semeru Runtimes.
This indeed solve my issue, and my core.xxx.dmp can now be read and have all internal attributes and thread stack.
If this can help other people, this are some differences between the MAT with dtfj open source of 2020 and the IBM's MAT
- version of DTFJ upgraded to 2025
- .ini options added to MAT to support Semeru : for example "-Dorg.eclipse.mat.dtfj.implementation=IBM Semeru Runtimes". I tried to add this option to my opensource MAT with DTFJ 2020, but it didn't change anything, so i guess the 2025 dtfj is the main difference.