ROCm-Developer-Tools
Standard output not being flushed during program execution when using Open MPI and rocprof
When running a MPI program with Open MPI and rocprof, all that has been written in the standard output is only flushed at the end of the program execution at once even if "\n" has been written in the standard output . When using MPICH and rocprof or Open MPI without rocprof, the standard output is automatically flushed during program execution when "\n" has been written in the standard output. When using the fflush function, the standard output is correctly flushed during program execution with Open MPI and rocprof.
This can be problematic for long programs that rely on printing "\n" in the standard output rather than using fflush(stdout) to display the progress of the program (machine learning for example).
A test case is provided attached to this issue. It can be build with :
mpicc openmpi_rocprof_issue.c -L/opt/rocm/hip/lib/ -l amdhip64 -I /opt/rocm/include -D __HIP_PLATFORM_AMD__
I ran it with only one rank to avoid post-processing issues :
mpirun -n 1 rocprof --hip-trace -d my_trace ./a.out
Line 13 can be commented/uncommented to check the behavior when using fflush(stdout)
You can try to specify line buffering for stdout explicitly
setvbuf(stdout, NULL, _IOLBF, 4096);
@yoann-heitz Do you still need assistance with this ticket? If not, please close the ticket. Thanks!
@yoann-heitz Closing ticket. Please feel free to re-open a ticket if you still see the issue with the latest ROCm. Thanks!