Memory OOM saving model.

Open llamazing opened this issue 6 years ago • 2 comments

It costs about 130G memory training model but when to save model it occurs oom error(total memory 195G). Is there extra memory allocation when saving the model?

Dec 21 '19 00:12 llamazing

There shouldn't be large memory allocation. Maybe it is caused by pickle?

We will check the memory usage later.

Dec 22 '19 02:12 KiddoZhu

There shouldn't be large memory allocation. Maybe it is caused by pickle?

We will check the memory usage later.

It is caused by pickle. I replace joblib with pickle. The joblib uses less memory when dumps model but it costs more disk capacity.

Dec 26 '19 03:12 llamazing