hi, I am trying to run train.py, I get an error as "TypeError: h5py objects cannot be pickled".
I think the error is stemming from torch.utils.data.DataLoader.
or did i miss any configurations?
i have no idea about how to tackle it.
the error stack trace is as follow.
Traceback (most recent call last):
File "D:/study/workspace/Python/a-PyTorch-Tutorial-to-Image-Captioning/train.py", line 333, in
main()
File "D:/study/workspace/Python/a-PyTorch-Tutorial-to-Image-Captioning/train.py", line 117, in main
epoch=epoch)
File "D:/study/workspace/Python/a-PyTorch-Tutorial-to-Image-Captioning/train.py", line 164, in train
for i, (imgs, caps, caplens) in enumerate(train_loader):
File "D:\Environment\Anaconda\envs\PyTorch\lib\site-packages\torch\utils\data\dataloader.py", line 278, in iter
return _MultiProcessingDataLoaderIter(self)
File "D:\Environment\Anaconda\envs\PyTorch\lib\site-packages\torch\utils\data\dataloader.py", line 682, in init
w.start()
File "D:\Environment\Anaconda\envs\PyTorch\lib\multiprocessing\process.py", line 112, in start
self._popen = self._Popen(self)
File "D:\Environment\Anaconda\envs\PyTorch\lib\multiprocessing\context.py", line 223, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "D:\Environment\Anaconda\envs\PyTorch\lib\multiprocessing\context.py", line 322, in _Popen
return Popen(process_obj)
File "D:\Environment\Anaconda\envs\PyTorch\lib\multiprocessing\popen_spawn_win32.py", line 89, in init
reduction.dump(process_obj, to_child)
File "D:\Environment\Anaconda\envs\PyTorch\lib\multiprocessing\reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
File "D:\Environment\Anaconda\envs\PyTorch\lib\site-packages\h5py_hl\base.py", line 308, in getnewargs
raise TypeError("h5py objects cannot be pickled")
TypeError: h5py objects cannot be pickled
Process finished with exit code 1
hello,I face the same problems with you? how can i handle this problems?
@liuyyy111 @Christine620
It happens because h5py won't read from multiple processes. By omitting num_workers, you're setting it to the default of 0, which uses only the main process. You could also set it to 1 (on Linux, at least).
So try to set :
num_workers=0.
it works for me.
sgrvinod
