Core dump during execution of Binary

[borosilicate]

In my cyber Security class we are doing a buffer overflow and attempting to cause the program to run a shell and I was really close to crafting the payload and it crashed and said my command I tried to execute was unrecognized then it had a core dump and I couldn't see what my payload was because the session ended. I was wondering if this is normal and how peda should run?