gflownet icon indicating copy to clipboard operation
gflownet copied to clipboard

Investigate multiprocessing memory usage further

Open bengioe opened this issue 2 years ago • 0 comments

It's not clear that we are doing multiprocessing in the best way possible.

  • we have been seeing problems with too many shared tensors being created, the current patch to that is #59 (which naively pickles everything!)
  • we obviously waste time copying tensors and objects from/to workers

A great enhancement for this project would be to dig deeper into those problems.

bengioe avatar Apr 26 '23 18:04 bengioe