sigma comments

Results 6 comments of


                                            sigma

IdEmbedding not support PyTorchEngine

Do you have some methods to get embedding-batch by input-batch

IdEmbedding not support PyTorchEngine

by using NDArrayEx Interface, I am already found a solution as follow. NDArray inputUserEmbedding = userInput.getNDArrayInternal() .embedding(userInput, userEmbeddingTable, SparseFormat.DENSE) .singletonOrThrow();

set epoch=500, gpus=2, runing reach epoch=194, jvm gc very high, gpu utility is 0%, and cpu reach 100%.

By My Test, GPU memory is very stable, but CPU memory grows bigger and bigger. epoch = 0, CPU memory just used 80GB, but when echo = 30, CPU memory...

[💡SUG] Will multi-GPU be supported in the next version?

I suggest you to learn the method of DJL (DeepJavaLibrary https://djl.ai/ ), it could use all the gpus of single machine.

[💡SUG] 对数据集的优化

是否考虑使用 https://github.com/h2oai/datatable.git 代替 pandas，或者使用pyspark代替pandas 充分利用cpu的能力，加速数据预处理。

import mgclient error

#conda activate xss #openssl version #OpenSSL 1.1.1t 7 Feb 2023 # python setup.py install successfully, but import mgclient, throw such error as above,