UnderstandingNLP icon indicating copy to clipboard operation
UnderstandingNLP copied to clipboard
verified publisher icon bhadreshpsavani

TFLite on GPUs

Open duongkstn opened this issue 4 years ago • 3 comments

Hi, your notebook TFLiteExperiments.ipynb are awesome, I can run it in my CPU (tensorflow 2.3.1 or 2.4.1 is ok) without any bugs. But have you tried:

  • run that notebook on a single GPU ? I tried but it failed as following:
INFO: TfLiteFlexDelegate delegate: 517 nodes delegated out of 517 nodes with 1 partitions.

Segmentation fault (core dumped)
  • run on CPU and GPU with input is a batch ?
  • run on CPU and GPU with input is a batch but with dynamic sequence length ?

duongkstn avatar May 06 '21 09:05 duongkstn

Hi,

Thanks for letting me know,

Yesterday only I come across this thing! Earlier when I run it on GPU, it was working fine. Because of some recent changes in Transformers I think this bug is introduced.

I will fix the transformer version in the colab notebook and it will be resolve.

bhadreshpsavani avatar May 06 '21 09:05 bhadreshpsavani

By the way, I am using transformers==3.5.1 in CPU and GPU. Your code works with tensorflow 2.3.1 and transformers 3.5.1 in CPU so I don't think version is an issue

duongkstn avatar May 06 '21 09:05 duongkstn

Ya, you are right!

I just tried in the colab notebook recently but it was crushing at inference, I will check and let you know if I found some solutions for running inference on GPU issue. Please also let me know If you discover any solution

bhadreshpsavani avatar May 06 '21 10:05 bhadreshpsavani