fairseq2 icon indicating copy to clipboard operation
fairseq2 copied to clipboard

Simple wrapper for read_iterator pickleability

Open syleshfb opened this issue 1 year ago • 0 comments

What does this PR do? Please describe: Adds a utility function, read_and_pickle_wrap_iterator, to which any iterator/generator can be passed. This will wrap the iterator/generator in a class whose pickle behavior is to track an internal counter for the number of times __next__() is called, and to replay this during reloading. If the given iterator is already pickleable, the wrapping is elided.

Does your PR introduce any breaking changes? If yes, please list them: N/A

Check list:

  • [ ] Was the content of this PR discussed and approved via a GitHub issue? (no need for typos or documentation improvements)
  • [ ] Did you read the contributor guideline?
  • [ ] Did you make sure that your PR does only one thing instead of bundling different changes together?
  • [ ] Did you make sure to update the documentation with your changes? (if necessary)
  • [ ] Did you write any new necessary tests?
  • [ ] Did you verify new and existing tests pass locally with your changes?
  • [ ] Did you update the CHANGELOG? (no need for typos, documentation, or minor internal changes)

syleshfb avatar Jul 30 '24 21:07 syleshfb