microsoft
Make game creation thread safe
At the moment, compile_game is not thread-safe. This is problematic when games are generated on-the-fly during batch training. For instance, see http://github.com/xingdi-eric-yuan/TextWorld-Coin-Collector/issues/3.
@tavianator do you know what would be the best way to ensure game creation (writing to disk) is thread-safe. I was thinking of using something like fcntl.lockf but we need to write multiple files (.json and .ni and .ulx) during game compilation? Also, we can assume identical games should have the same name (i.e. same UUID).
File locking (flock(), lockf(), etc.) is rarely the right thing.
What's the actual conflict? Are multiple threads generating the same game at the same time? If it's just that then the regular "atomic write" pattern is enough, where you write to a temporary file and then rename it once it's done. E.g. https://github.com/untitaker/python-atomicwrites
Yes, you are correct, multiple processes might be generating the same game at the same time. I used to use a temporary folder as an atomic write operation. I think I was using a folder rather than a file because I was on a distributed filesystem (like Lustre). Thanks, I'll check the link out.
Renaming the folder could work too, it's a bit different though. If you lose the race to rename the folder, the rename will fail (unlike overwriting files which succeeds).
compile_game doesn't have to be thread-safe. We recommend generating the games in a temporary folder, then move the generated file afterward.