Benjamin Moore Wagman

Suspect (incorrectly--see subsequent comments) that error was caused by writing simulation output to /cfs instead of /pscratch.

To complicate things, I just ran successfully on /cfs for the first time: `/global/cfs/cdirs/e3sm/emulate/E3SM_simulations/20230710.replicate.F2010.v3atm_on_master.pm-cpu.8.cfs` I'll re-run the failed case mentioned when I opened the issue: ` /global/cfs/cdirs/e3sm/emulate/E3SM_simulations/replicate.F2010.v3atm_on_master.pm-cpu/` as ` /global/cfs/cdirs/e3sm/emulate/E3SM_simulations/replicate.F2010.v3atm_on_master.pm-cpu.retry/`....

@wlin7, thank you for confirming the error on pm-cpu on /pscratch. I will try pm-cpu_intel. I hope we can get to the bottom of this soon.

I also replicated the SNOWDP error on pm-cpu writing to /pscratch with gnu compiler. I was not able to set up a case using the intel compiler but will keep...

I have not seen the crash since then, but I have not run enough simulations to answer whether it can be closed.