testcorenrn_patstim::coreneuron_gpu_offline test does not produce reliable output

Open olupton opened this issue 4 years ago • 1 comments
Describe the issue For several days we have seen hard-to-reproduce CI errors when the testcorenrn_patstim::coreneuron_gpu_offline test in the NEURON CTest suite produces output spikes that do not match NEURON or CoreNEURON with a CPU backend. The failing tests were run with 2 MPI ranks, reducing this to 1 seems to avoid the problem.
To Reproduce Build NEURON + CoreNEURON-with-GPU-support, make sure the testcorenrn_patstim::coreneuron_gpu_offline test is using 2 MPI ranks...see if the CTest suite passes. Reproducing this locally has been hard, the CI environment seems to be more prone to it.
Because the CoreNEURON+GPU configuration does not crash, the test that actually fails is testcorenrn_patstim::compare_results, which compares the spike outputs from different configurations.
Expected behavior CoreNEURON+GPU should consistent match NEURON/CoreNEURON+CPU/reference file.
System (please complete the following information)
OS: BB5
Compiler: NVHPC 21.2
Version: master
Backend: GPU
May 19 '21 15:05 olupton
This test failed in a CI run for #574 with only one MPI rank and the following error:
37: (null) lives at 0x2a5db80 size 128 partially present
37: Present table dump for device[1]: NVIDIA Tesla GPU 0, compute capability 7.0, threadid=1
37: host:0xad3980 device:0x2acae3e02e00 size:8 presentcount:0+1 line:-1 name:_ZN10coreneuron12erev_hhwatchE
37: host:0xad3988 device:0x2acae3e02f00 size:8 presentcount:0+1 line:-1 name:_ZN10coreneuron10ek_hhwatchE
37: host:0xad3990 device:0x2acae3e03000 size:8 presentcount:0+1 line:-1 name:_ZN10coreneuron11ena_hhwatchE
37: host:0xad3998 device:0x2acae3e03100 size:8 presentcount:0+1 line:-1 name:_ZN10coreneuron12gpas_hhwatchE
37: host:0xad39a0 device:0x2acae3e03200 size:8 presentcount:0+1 line:-1 name:_ZN10coreneuron10gk_hhwatchE
37: host:0xad39a8 device:0x2acae3e03300 size:8 presentcount:0+1 line:-1 name:_ZN10coreneuron11gna_hhwatchE
37: host:0xad4170 device:0x2acae3e03400 size:8 presentcount:0+1 line:-1 name:_ZN10coreneuron9nabath_naE
37: host:0xad41e0 device:0x2acae3e03500 size:8 presentcount:0+1 line:-1 name:_ZN10coreneuron10FARADAY_naE
37: host:0xad41e8 device:0x2acae3e03600 size:8 presentcount:0+1 line:-1 name:_ZN10coreneuron4R_naE
37: host:0xbf41a0 device:0x2acae3e00f00 size:4 presentcount:0+1 line:-1 name:_ZN10coreneuron17_mechtype_Gfluct3E
37: host:0xbf41f8 device:0x2acae3e01200 size:8 presentcount:0+1 line:-1 name:_st__gpfs_bbp_cscs_ch_ssd_gitlab_map_jobs_bbpcihpcproj12_P7466_24805_spack_build_spack_stage_neuron_develop_zdz3jitzh6wukjgutc36fji6dtpkm7g2_spack_build_zdz3jit_test_nrnivmodl_16e59b413722fa2d993abf06fc2e64a772692803ff5913bd0eb34e9c8bd45725_x86_64_corenrn_mod2c_exp2syn_cpp__ZN54_INTERNAL_32_x86_64_corenrn_mod2c_exp2syn_cpp_25c7f1d810coreneuron2A0E
37: host:0xbf4200 device:0x2acae3e01300 size:8 presentcount:0+1 line:-1 name:_st__gpfs_bbp_cscs_ch_ssd_gitlab_map_jobs_bbpcihpcproj12_P7466_24805_spack_build_spack_stage_neuron_develop_zdz3jitzh6wukjgutc36fji6dtpkm7g2_spack_build_zdz3jit_test_nrnivmodl_16e59b413722fa2d993abf06fc2e64a772692803ff5913bd0eb34e9c8bd45725_x86_64_corenrn_mod2c_exp2syn_cpp__ZN54_INTERNAL_32_x86_64_corenrn_mod2c_exp2syn_cpp_25c7f1d810coreneuron2B0E
37: host:0xbf4238 device:0x2acae3e01000 size:8 presentcount:0+1 line:-1 name:_ZN10coreneuron15_slist1_Exp2SynE
37: host:0xbf4240 device:0x2acae3e01100 size:8 presentcount:0+1 line:-1 name:_ZN10coreneuron15_dlist1_Exp2SynE
37: host:0xbf4278 device:0x2acae3e01600 size:8 presentcount:0+1 line:-1 name:_st__gpfs_bbp_cscs_ch_ssd_gitlab_map_jobs_bbpcihpcproj12_P7466_24805_spack_build_spack_stage_neuron_develop_zdz3jitzh6wukjgutc36fji6dtpkm7g2_spack_build_zdz3jit_test_nrnivmodl_16e59b413722fa2d993abf06fc2e64a772692803ff5913bd0eb34e9c8bd45725_x86_64_corenrn_mod2c_expsyn_cpp__ZN53_INTERNAL_31_x86_64_corenrn_mod2c_expsyn_cpp_0127273610coreneuron2g0E
37: host:0xbf42a8 device:0x2acae3e01400 size:8 presentcount:0+1 line:-1 name:_ZN10coreneuron14_slist1_ExpSynE
37: host:0xbf42b0 device:0x2acae3e01500 size:8 presentcount:0+1 line:-1 name:_ZN10coreneuron14_dlist1_ExpSynE
37: host:0xbf42e8 device:0x2acae3e01700 size:8 presentcount:0+1 line:-1 name:_ZN10coreneuron12_celsius__hhE
37: host:0xbf42f0 device:0x2acae3e01a00 size:8 presentcount:0+1 line:-1 name:_st__gpfs_bbp_cscs_ch_ssd_gitlab_map_jobs_bbpcihpcproj12_P7466_24805_spack_build_spack_stage_neuron_develop_zdz3jitzh6wukjgutc36fji6dtpkm7g2_spack_build_zdz3jit_test_nrnivmodl_16e59b413722fa2d993abf06fc2e64a772692803ff5913bd0eb34e9c8bd45725_x86_64_corenrn_mod2c_hh_cpp__ZN49_INTERNAL_27_x86_64_corenrn_mod2c_hh_cpp_4898369610coreneuron2h0E
37: host:0xbf42f8 device:0x2acae3e01b00 size:8 presentcount:0+1 line:-1 name:_st__gpfs_bbp_cscs_ch_ssd_gitlab_map_jobs_bbpcihpcproj12_P7466_24805_spack_build_spack_stage_neuron_develop_zdz3jitzh6wukjgutc36fji6dtpkm7g2_spack_build_zdz3jit_test_nrnivmodl_16e59b413722fa2d993abf06fc2e64a772692803ff5913bd0eb34e9c8bd45725_x86_64_corenrn_mod2c_hh_cpp__ZN49_INTERNAL_27_x86_64_corenrn_mod2c_hh_cpp_4898369610coreneuron2m0E
37: host:0xbf4300 device:0x2acae3e01c00 size:8 presentcount:0+1 line:-1 name:_st__gpfs_bbp_cscs_ch_ssd_gitlab_map_jobs_bbpcihpcproj12_P7466_24805_spack_build_spack_stage_neuron_develop_zdz3jitzh6wukjgutc36fji6dtpkm7g2_spack_build_zdz3jit_test_nrnivmodl_16e59b413722fa2d993abf06fc2e64a772692803ff5913bd0eb34e9c8bd45725_x86_64_corenrn_mod2c_hh_cpp__ZN49_INTERNAL_27_x86_64_corenrn_mod2c_hh_cpp_4898369610coreneuron2n0E
37: host:0xbf4338 device:0x2acae3e01800 size:8 presentcount:0+1 line:-1 name:_ZN10coreneuron10_slist1_hhE
37: host:0xbf4340 device:0x2acae3e01900 size:8 presentcount:0+1 line:-1 name:_ZN10coreneuron10_dlist1_hhE
37: host:0xbf4378 device:0x2acae3e01d00 size:8 presentcount:0+1 line:-1 name:_ZN10coreneuron17_celsius__hhderivE
37: host:0xbf4380 device:0x2acae3e02100 size:8 presentcount:0+1 line:-1 name:_st__gpfs_bbp_cscs_ch_ssd_gitlab_map_jobs_bbpcihpcproj12_P7466_24805_spack_build_spack_stage_neuron_develop_zdz3jitzh6wukjgutc36fji6dtpkm7g2_spack_build_zdz3jit_test_nrnivmodl_16e59b413722fa2d993abf06fc2e64a772692803ff5913bd0eb34e9c8bd45725_x86_64_corenrn_mod2c_hhderiv_cpp__ZN54_INTERNAL_32_x86_64_corenrn_mod2c_hhderiv_cpp_14e5a1e310coreneuron2h0E
37: host:0xbf4388 device:0x2acae3e02200 size:8 presentcount:0+1 line:-1 name:_st__gpfs_bbp_cscs_ch_ssd_gitlab_map_jobs_bbpcihpcproj12_P7466_24805_spack_build_spack_stage_neuron_develop_zdz3jitzh6wukjgutc36fji6dtpkm7g2_spack_build_zdz3jit_test_nrnivmodl_16e59b413722fa2d993abf06fc2e64a772692803ff5913bd0eb34e9c8bd45725_x86_64_corenrn_mod2c_hhderiv_cpp__ZN54_INTERNAL_32_x86_64_corenrn_mod2c_hhderiv_cpp_14e5a1e310coreneuron2m0E
37: host:0xbf4390 device:0x2acae3e02300 size:8 presentcount:0+1 line:-1 name:_st__gpfs_bbp_cscs_ch_ssd_gitlab_map_jobs_bbpcihpcproj12_P7466_24805_spack_build_spack_stage_neuron_develop_zdz3jitzh6wukjgutc36fji6dtpkm7g2_spack_build_zdz3jit_test_nrnivmodl_16e59b413722fa2d993abf06fc2e64a772692803ff5913bd0eb34e9c8bd45725_x86_64_corenrn_mod2c_hhderiv_cpp__ZN54_INTERNAL_32_x86_64_corenrn_mod2c_hhderiv_cpp_14e5a1e310coreneuron2n0E
37: host:0xbf43c8 device:0x2acae3e01e00 size:8 presentcount:0+1 line:-1 name:_ZN10coreneuron15_slist2_hhderivE
37: host:0xbf43d0 device:0x2acae3e01f00 size:8 presentcount:0+1 line:-1 name:_ZN10coreneuron15_slist1_hhderivE
37: host:0xbf43d8 device:0x2acae3e02000 size:8 presentcount:0+1 line:-1 name:_ZN10coreneuron15_dlist1_hhderivE
37: host:0xbf4418 device:0x2acae3e02400 size:8 presentcount:0+1 line:-1 name:_ZN10coreneuron15_celsius__hhkinE
37: host:0xbf4420 device:0x2acae3e02700 size:8 presentcount:0+1 line:-1 name:_st__gpfs_bbp_cscs_ch_ssd_gitlab_map_jobs_bbpcihpcproj12_P7466_24805_spack_build_spack_stage_neuron_develop_zdz3jitzh6wukjgutc36fji6dtpkm7g2_spack_build_zdz3jit_test_nrnivmodl_16e59b413722fa2d993abf06fc2e64a772692803ff5913bd0eb34e9c8bd45725_x86_64_corenrn_mod2c_hhkin_cpp__ZN52_INTERNAL_30_x86_64_corenrn_mod2c_hhkin_cpp_8358a5b710coreneuron3hc0E
37: host:0xbf4428 device:0x2acae3e02800 size:8 presentcount:0+1 line:-1 name:_st__gpfs_bbp_cscs_ch_ssd_gitlab_map_jobs_bbpcihpcproj12_P7466_24805_spack_build_spack_stage_neuron_develop_zdz3jitzh6wukjgutc36fji6dtpkm7g2_spack_build_zdz3jit_test_nrnivmodl_16e59b413722fa2d993abf06fc2e64a772692803ff5913bd0eb34e9c8bd45725_x86_64_corenrn_mod2c_hhkin_cpp__ZN52_INTERNAL_30_x86_64_corenrn_mod2c_hhkin_cpp_8358a5b710coreneuron2h0E
37: host:0xbf4430 device:0x2acae3e02900 size:8 presentcount:0+1 line:-1 name:_st__gpfs_bbp_cscs_ch_ssd_gitlab_map_jobs_bbpcihpcproj12_P7466_24805_spack_build_spack_stage_neuron_develop_zdz3jitzh6wukjgutc36fji6dtpkm7g2_spack_build_zdz3jit_test_nrnivmodl_16e59b413722fa2d993abf06fc2e64a772692803ff5913bd0eb34e9c8bd45725_x86_64_corenrn_mod2c_hhkin_cpp__ZN52_INTERNAL_30_x86_64_corenrn_mod2c_hhkin_cpp_8358a5b710coreneuron3mc0E
37: host:0xbf4438 device:0x2acae3e02a00 size:8 presentcount:0+1 line:-1 name:_st__gpfs_bbp_cscs_ch_ssd_gitlab_map_jobs_bbpcihpcproj12_P7466_24805_spack_build_spack_stage_neuron_develop_zdz3jitzh6wukjgutc36fji6dtpkm7g2_spack_build_zdz3jit_test_nrnivmodl_16e59b413722fa2d993abf06fc2e64a772692803ff5913bd0eb34e9c8bd45725_x86_64_corenrn_mod2c_hhkin_cpp__ZN52_INTERNAL_30_x86_64_corenrn_mod2c_hhkin_cpp_8358a5b710coreneuron2m0E
37: host:0xbf4440 device:0x2acae3e02b00 size:8 presentcount:0+1 line:-1 name:_st__gpfs_bbp_cscs_ch_ssd_gitlab_map_jobs_bbpcihpcproj12_P7466_24805_spack_build_spack_stage_neuron_develop_zdz3jitzh6wukjgutc36fji6dtpkm7g2_spack_build_zdz3jit_test_nrnivmodl_16e59b413722fa2d993abf06fc2e64a772692803ff5913bd0eb34e9c8bd45725_x86_64_corenrn_mod2c_hhkin_cpp__ZN52_INTERNAL_30_x86_64_corenrn_mod2c_hhkin_cpp_8358a5b710coreneuron3nc0E
37: host:0xbf4448 device:0x2acae3e02c00 size:8 presentcount:0+1 line:-1 name:_st__gpfs_bbp_cscs_ch_ssd_gitlab_map_jobs_bbpcihpcproj12_P7466_24805_spack_build_spack_stage_neuron_develop_zdz3jitzh6wukjgutc36fji6dtpkm7g2_spack_build_zdz3jit_test_nrnivmodl_16e59b413722fa2d993abf06fc2e64a772692803ff5913bd0eb34e9c8bd45725_x86_64_corenrn_mod2c_hhkin_cpp__ZN52_INTERNAL_30_x86_64_corenrn_mod2c_hhkin_cpp_8358a5b710coreneuron2n0E
37: host:0xbf4478 device:0x2acae3e02500 size:8 presentcount:0+1 line:-1 name:_ZN10coreneuron13_slist1_hhkinE
37: host:0xbf4480 device:0x2acae3e02600 size:8 presentcount:0+1 line:-1 name:_ZN10coreneuron13_dlist1_hhkinE
37: host:0xbf44b0 device:0x2acae3e02d00 size:4 presentcount:0+1 line:-1 name:_ZN10coreneuron17_mechtype_hhwatchE
37: host:0xbf4518 device:0x2acae3e03700 size:8 presentcount:0+1 line:-1 name:_ZN10coreneuron10_slist1_naE
37: host:0xbf4520 device:0x2acae3e03800 size:8 presentcount:0+1 line:-1 name:_ZN10coreneuron10_dlist1_naE
37: host:0xbf6030 device:0x2acae3e03a00 size:4 presentcount:0+1 line:-1 name:_ZN10coreneuron11secondorderE
37: host:0xbf6048 device:0x2acae3e03900 size:8 presentcount:0+1 line:-1 name:_ZN10coreneuron7celsiusE
37: host:0xbf6050 device:0x2acae3e03b00 size:8 presentcount:0+1 line:-1 name:_ZN10coreneuron2piE
37: host:0x2918980 device:0x2acae3afb000 size:12 presentcount:0+1 line:728 name:_ZN10coreneuron15_slist2_hhderivE
37: host:0x2919420 device:0x2acae3afb200 size:24 presentcount:0+1 line:788 name:_ZN10coreneuron13_slist1_hhkinE
37: host:0x2919440 device:0x2acae3afb400 size:24 presentcount:0+1 line:788 name:_ZN10coreneuron13_dlist1_hhkinE
37: host:0x291f6c0 device:0x2acae3afb800 size:8 presentcount:0+1 line:480 name:_ZN10coreneuron10_dlist1_naE
37: host:0x2941360 device:0x2acae3afb600 size:8 presentcount:0+1 line:480 name:_ZN10coreneuron10_slist1_naE
37: host:0x2941470 device:0x2acae3afac00 size:12 presentcount:0+1 line:722 name:_ZN10coreneuron15_slist1_hhderivE
37: host:0x2941490 device:0x2acae3afae00 size:12 presentcount:0+1 line:722 name:_ZN10coreneuron15_dlist1_hhderivE
37: host:0x2950f30 device:0x2acae3afa400 size:4 presentcount:0+1 line:606 name:_ZN10coreneuron14_slist1_ExpSynE
37: host:0x2950f50 device:0x2acae3afa600 size:4 presentcount:0+1 line:606 name:_ZN10coreneuron14_dlist1_ExpSynE
37: host:0x296ca80 device:0x2acae3afa000 size:12 presentcount:0+1 line:618 name:_ZN10coreneuron10_slist1_hhE
37: host:0x296caa0 device:0x2acae3afa200 size:12 presentcount:0+1 line:618 name:_ZN10coreneuron10_dlist1_hhE
37: host:0x2990be0 device:0x2acae3afbe00 size:24 presentcount:0+1 line:-1 name:(null)
37: host:0x29a2810 device:0x2acae3afba00 size:136 presentcount:0+1 line:-1 name:(null)
37: host:0x29a28a0 device:0x2acae3afbc00 size:24 presentcount:0+1 line:-1 name:(null)
37: host:0x2a138b0 device:0x2acae3afa800 size:8 presentcount:0+1 line:639 name:_ZN10coreneuron15_slist1_Exp2SynE
37: host:0x2a138d0 device:0x2acae3afaa00 size:8 presentcount:0+1 line:639 name:_ZN10coreneuron15_dlist1_Exp2SynE
37: host:0x2a52338 device:0x2acae3afc000 size:912 presentcount:0+1 line:-1 name:(null)
37: host:0x2a5ae00 device:0x2acae3afe200 size:120 presentcount:0+1 line:-1 name:(null)
37: host:0x2a5ae80 device:0x2acae3afc400 size:7296 presentcount:0+1 line:-1 name:(null)
37: host:0x2a5ce40 device:0x2acae3afe400 size:240 presentcount:0+1 line:-1 name:(null)
37: host:0x2a5d480 device:0x2acae3afe600 size:40 presentcount:0+1 line:-1 name:(null)
37: host:0x2a5d540 device:0x2acae3afe800 size:72 presentcount:0+1 line:-1 name:(null)
37: host:0x2a5d600 device:0x2acae3afec00 size:40 presentcount:0+1 line:-1 name:(null)
37: host:0x2a5d650 device:0x2acae3b01a00 size:40 presentcount:0+1 line:-1 name:(null)
37: host:0x2a5d680 device:0x2acae3afee00 size:72 presentcount:0+1 line:-1 name:(null)
37: host:0x2a5d740 device:0x2acae3aff400 size:40 presentcount:0+1 line:-1 name:(null)
37: host:0x2a5d800 device:0x2acae3aff600 size:72 presentcount:0+1 line:-1 name:(null)
37: host:0x2a5d8c0 device:0x2acae3affc00 size:40 presentcount:0+1 line:-1 name:(null)
37: host:0x2a5d940 device:0x2acae3affe00 size:72 presentcount:0+1 line:-1 name:(null)
37: host:0x2a5da00 device:0x2acae3b01200 size:40 presentcount:0+1 line:-1 name:(null)
37: host:0x2a5dac0 device:0x2acae3b01400 size:72 presentcount:0+1 line:-1 name:(null)
37: host:0x2a5db50 device:0x2acae3b01e00 size:128 presentcount:0+1 line:-1 name:(null)
37: host:0x2a5dc30 device:0x2acae3b01c00 size:72 presentcount:0+1 line:-1 name:(null)
37: host:0x2a5dfc0 device:0x2acae3afea00 size:40 presentcount:0+1 line:-1 name:(null)
37: host:0x2a5e040 device:0x2acae3aff000 size:40 presentcount:0+1 line:-1 name:(null)
37: host:0x2a5e0c0 device:0x2acae3aff200 size:64 presentcount:0+1 line:-1 name:(null)
37: host:0x2a5e180 device:0x2acae3aff800 size:40 presentcount:0+1 line:-1 name:(null)
37: host:0x2a5e200 device:0x2acae3affa00 size:64 presentcount:0+1 line:-1 name:(null)
37: host:0x2a5e2c0 device:0x2acae3b00000 size:40 presentcount:0+1 line:-1 name:(null)
37: host:0x2a5e340 device:0x2acae3b00200 size:128 presentcount:0+1 line:-1 name:(null)
37: host:0x2a5e440 device:0x2acae3b01600 size:40 presentcount:0+1 line:-1 name:(null)
37: host:0x2a5e4c0 device:0x2acae3b01800 size:384 presentcount:0+1 line:-1 name:(null)
37: host:0x2a5ed00 device:0x2acae3b00400 size:64 presentcount:0+1 line:-1 name:(null)
37: host:0x2a5edc0 device:0x2acae3b00600 size:40 presentcount:0+1 line:-1 name:(null)
37: host:0x2a5ee40 device:0x2acae3b00e00 size:44 presentcount:0+1 line:-1 name:(null)
37: host:0x2a5eec0 device:0x2acae3b01000 size:44 presentcount:0+1 line:-1 name:(null)
37: host:0x2a5ef80 device:0x2acae3b00800 size:40 presentcount:0+1 line:-1 name:(null)
37: host:0x2a5f000 device:0x2acae3b00a00 size:80 presentcount:0+1 line:-1 name:(null)
37: host:0x2a5f100 device:0x2acae3b00c00 size:80 presentcount:0+1 line:-1 name:(null)
37: allocated block device:0x2acae3afa000 size:512 thread:1
37: allocated block device:0x2acae3afa200 size:512 thread:1
37: allocated block device:0x2acae3afa400 size:512 thread:1
37: allocated block device:0x2acae3afa600 size:512 thread:1
37: allocated block device:0x2acae3afa800 size:512 thread:1
37: allocated block device:0x2acae3afaa00 size:512 thread:1
37: allocated block device:0x2acae3afac00 size:512 thread:1
37: allocated block device:0x2acae3afae00 size:512 thread:1
37: allocated block device:0x2acae3afb000 size:512 thread:1
37: allocated block device:0x2acae3afb200 size:512 thread:1
37: allocated block device:0x2acae3afb400 size:512 thread:1
37: allocated block device:0x2acae3afb600 size:512 thread:1
37: allocated block device:0x2acae3afb800 size:512 thread:1
37: allocated block device:0x2acae3afba00 size:512 thread:1
37: allocated block device:0x2acae3afbc00 size:512 thread:1
37: allocated block device:0x2acae3afbe00 size:512 thread:1
37: allocated block device:0x2acae3afc000 size:1024 thread:1
37: allocated block device:0x2acae3afc400 size:7680 thread:1
37: allocated block device:0x2acae3afe200 size:512 thread:1
37: allocated block device:0x2acae3afe400 size:512 thread:1
37: allocated block device:0x2acae3afe600 size:512 thread:1
37: allocated block device:0x2acae3afe800 size:512 thread:1
37: allocated block device:0x2acae3afea00 size:512 thread:1
37: allocated block device:0x2acae3afec00 size:512 thread:1
37: allocated block device:0x2acae3afee00 size:512 thread:1
37: allocated block device:0x2acae3aff000 size:512 thread:1
37: allocated block device:0x2acae3aff200 size:512 thread:1
37: allocated block device:0x2acae3aff400 size:512 thread:1
37: allocated block device:0x2acae3aff600 size:512 thread:1
37: allocated block device:0x2acae3aff800 size:512 thread:1
37: allocated block device:0x2acae3affa00 size:512 thread:1
37: allocated block device:0x2acae3affc00 size:512 thread:1
37: allocated block device:0x2acae3affe00 size:512 thread:1
37: allocated block device:0x2acae3b00000 size:512 thread:1
37: allocated block device:0x2acae3b00200 size:512 thread:1
37: allocated block device:0x2acae3b00400 size:512 thread:1
37: allocated block device:0x2acae3b00600 size:512 thread:1
37: allocated block device:0x2acae3b00800 size:512 thread:1
37: allocated block device:0x2acae3b00a00 size:512 thread:1
37: allocated block device:0x2acae3b00c00 size:512 thread:1
37: allocated block device:0x2acae3b00e00 size:512 thread:1
37: allocated block device:0x2acae3b01000 size:512 thread:1
37: allocated block device:0x2acae3b01200 size:512 thread:1
37: allocated block device:0x2acae3b01400 size:512 thread:1
37: allocated block device:0x2acae3b01600 size:512 thread:1
37: allocated block device:0x2acae3b01800 size:512 thread:1
37: allocated block device:0x2acae3b01a00 size:512 thread:1
37: allocated block device:0x2acae3b01c00 size:512 thread:1
37: allocated block device:0x2acae3b01e00 size:512 thread:1
37: FATAL ERROR: variable in data clause is partially present on the device: name=(unknown)
37:  file:(OpenACC API) acc_copyin line:-1
this may be a red herring, but pasting it here anyway because information about this issue was rather scarce.
Jun 01 '21 09:06 olupton