maize
maize copied to clipboard
Fix cleanup_processes ValueError on normal workflow exit
Problem
Currently both examples and my own implementations end with a ValueError due to the dual cleanup mechanism in RunPool. The class registers cleanup_processes with atexit for safety, but also explicitly calls cleanup_processes in __exit__. This causes:
-
First cleanup: (
__exit__) successfully cleans up all processes, empties theprocslist -
Second cleanup: (
atexit) attempts to clean up again withitemslist intact butprocslist empty -
Result:
zip(items, procs, strict=True)raisesValueErrorbecause lists have different lengths in second cleanup
Solution
This PR modifies cleanup_processes to handle the dual cleanup scenario gracefully while improving safety:
- Early return for empty process list, skip cleanup when no processes remain
- Use
strict=Falseto allow standard cleanup in scenarios whereitemsis longer thanprocs - Orphan process detection, continues with process cleanup when more processes than items and raises
RuntimeErrorafterwards to indicate the mismatch
Testing
Adds a test suite (test_cleanup.py) covering:
- Normal case: Equal length lists (current behavior maintained)
- Dual cleanup case: Fewer processes than items (cleans up remaining processes without error)
- Orphan processes: More processes than items (kills process and raises ``RuntimeError```)
Thanks for maintaining this project! Happy to make any adjustments based on feedback.