omnitrace icon indicating copy to clipboard operation
omnitrace copied to clipboard

Segfault instrumenting Cray MPI w/ v1.2.0

Open skyreflectedinmirrors opened this issue 3 years ago • 2 comments

To reproduce on Crusher:

source sw/omnitrace-devel/share/omnitrace/setup-env.sh
module load craype-accel-amd-gfx90a
module load PrgEnv-cray
module load rocm
omnitrace -o $(basename /opt/cray/pe/lib64/libmpi_cray.so.12) -v 3 -- /opt/cray/pe/lib64/libmpi_cray.so.12
...
<output in attached log>

Looking at the core file shows:

(gdb) bt
#0  0x00007fffed4ef26f in Dyninst::Relocation::Instrumenter::handleCondDirExits(Dyninst::Relocation::RelocBlock*, Dyninst::Relocation::RelocGraph*, instPoint*) () from /autofs/nccs-svm1_home1/nicurtis/sw/omnitrace-devel/bin/../lib/omnitrace/libdyninstAPI.so.11.0
#1  0x00007fffed4f0015 in Dyninst::Relocation::Instrumenter::funcExitInstrumentation(Dyninst::Relocation::RelocBlock*, Dyninst::Relocation::RelocGraph*) () from /autofs/nccs-svm1_home1/nicurtis/sw/omnitrace-devel/bin/../lib/omnitrace/libdyninstAPI.so.11.0
#2  0x00007fffed4f020b in Dyninst::Relocation::Instrumenter::process(Dyninst::Relocation::RelocBlock*, Dyninst::Relocation::RelocGraph*) () from /autofs/nccs-svm1_home1/nicurtis/sw/omnitrace-devel/bin/../lib/omnitrace/libdyninstAPI.so.11.0
#3  0x00007fffed4ed280 in Dyninst::Relocation::Transformer::processGraph(Dyninst::Relocation::RelocGraph*) () from /autofs/nccs-svm1_home1/nicurtis/sw/omnitrace-devel/bin/../lib/omnitrace/libdyninstAPI.so.11.0
#4  0x00007fffed4d8c32 in Dyninst::Relocation::CodeMover::transform(Dyninst::Relocation::Transformer&) () from /autofs/nccs-svm1_home1/nicurtis/sw/omnitrace-devel/bin/../lib/omnitrace/libdyninstAPI.so.11.0
#5  0x00007fffed45cb4b in AddressSpace::transform(boost::shared_ptr<Dyninst::Relocation::CodeMover>) () from /autofs/nccs-svm1_home1/nicurtis/sw/omnitrace-devel/bin/../lib/omnitrace/libdyninstAPI.so.11.0
#6  0x00007fffed45dcf3 in AddressSpace::relocateInt(std::_Rb_tree_const_iterator<func_instance*>, std::_Rb_tree_const_iterator<func_instance*>, unsigned long) () from /autofs/nccs-svm1_home1/nicurtis/sw/omnitrace-devel/bin/../lib/omnitrace/libdyninstAPI.so.11.0
#7  0x00007fffed461fce in AddressSpace::relocate() () from /autofs/nccs-svm1_home1/nicurtis/sw/omnitrace-devel/bin/../lib/omnitrace/libdyninstAPI.so.11.0
#8  0x00007fffed506e1a in Dyninst::PatchAPI::DynInstrumenter::run() () from /autofs/nccs-svm1_home1/nicurtis/sw/omnitrace-devel/bin/../lib/omnitrace/libdyninstAPI.so.11.0
#9  0x00007fffed14f831 in Dyninst::PatchAPI::Patcher::run() () from /autofs/nccs-svm1_home1/nicurtis/sw/omnitrace-devel/bin/../lib/omnitrace/libpatchAPI.so.11.0
#10 0x00007fffed14f010 in Dyninst::PatchAPI::Command::commit() () from /autofs/nccs-svm1_home1/nicurtis/sw/omnitrace-devel/bin/../lib/omnitrace/libpatchAPI.so.11.0
#11 0x00007fffed45e97c in AddressSpace::patch(AddressSpace*) () from /autofs/nccs-svm1_home1/nicurtis/sw/omnitrace-devel/bin/../lib/omnitrace/libdyninstAPI.so.11.0
#12 0x00007fffed429c7e in BPatch_binaryEdit::writeFile(char const*) () from /autofs/nccs-svm1_home1/nicurtis/sw/omnitrace-devel/bin/../lib/omnitrace/libdyninstAPI.so.11.0
#13 0x000000000042670f in ?? ()
#14 0x00007fffe86ec2bd in __libc_start_main () from /lib64/libc.so.6
#15 0x00000000004299ea in ?? ()

skyreflectedinmirrors avatar Jul 18 '22 13:07 skyreflectedinmirrors

https://gist.github.com/arghdos/1132b39c116cc7dbe46a89b71b831274

skyreflectedinmirrors avatar Jul 18 '22 13:07 skyreflectedinmirrors

Yeah looks like another internal dyninst bug. I'll look into this when I'm working on #91

jrmadsen avatar Jul 18 '22 21:07 jrmadsen

This should be fixed by the Dyninst updates in #164

jrmadsen avatar Oct 21 '22 14:10 jrmadsen