lda2vec preprocess not good support about memory

Open hxsnow10 opened this issue 8 years ago • 1 comments

when i use 6G pure text， 3 threads, max_len=4W, they run out my 120Gmeory+180Gswap.

i guess may be some point unfriendly with big data.

Oct 10 '17 08:10 hxsnow10

I've encountered MemoryError several times and now have to fall back to a smaller amount of data.

Feb 17 '19 21:02 ghost