MPI AllReduce error

[abidmalikwaterloo]

I got the following errors

2018-07-16 15:27:27.536541: W tensorflow/core/framework/op_kernel.cc:1192] Unknown: Exception: Message truncated, error stack: MPI_Allreduce(855)..................: MPI_Allreduce(sbuf=MPI_IN_PLACE, rbuf=0x2049aaa00, count=256, MPI_FLOAT, MPI_SUM, MPI_COMM_WORLD) failed MPIR_Allreduce_impl(712)............: MPIR_Allreduce_intra(357)...........: MPIC_Sendrecv(186)..................: MPIDI_CH3U_Request_unpack_uebuf(599): Message truncated; 1536 bytes received but buffer size is 1024

Any comments!