Vaibhav Hiwase

icon indicating a website link https://vaibhavhiwase.webnode.com/ icon indicating an email [email protected]

icon company Celebal Technologies icon location Nagpur,Maharashtra,India icon location I am Technical Lead AI/ML in TechnoMile. I have a master's degree in Computer Science and Engineering. I love to contribute to open source.

Results 4 comments of Vaibhav Hiwase

CUDA out of Memory with low Memory Utilization (CUDA error: device-side assert triggered)

Please check {{management_address}}/models/ endpoint and monitor the following "jobQueueStatus": { "remainingCapacity": 100, "pendingRequests": 0 } I found this issue appears randomly when pendingRequests does not increases.

Update in inference.py -- omitting the optimizer's state

@microsoft-github-policy-service agree

Exception: CUDA error: an illegal memory access was encountered. CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.

@danielhanchen Apologies for the delay in responding. I'm currently testing the model with results obtained from OCR processing using Azure Document Intelligence. The inputs consist of random chunks of text...

Exception: CUDA error: an illegal memory access was encountered. CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.

@danielhanchen You are correct that we trained the model on Amazon EC2 G6 Instances, and inference is working fine there. However, we hosted the model inference on a different machine—specifically,...