ggerganov
Vulkan backend crashes with error DeviceLost
I built whisper.cpp with vulkan on android via Termux and I constantly get the same error:
libc++abi: terminating due to uncaught exception of type vk::DeviceLostError: vk::Queue::submit: ErrorDeviceLost
When running on tiny model, radically reducing audio context argument helps, but transcription results are obviously unpleasant. On larger models this has no effect.
Below are full logs from vulkaninfo and running running whisper-cli
same problem here...
System: Debian Linux 12 (kernel 6.1.0-40-amd64) Whisper.cpp-Version: 1.7.6 libvulkan-Version: 1.3.239 Mesa-Version: 22.3.6 Using mesa-vulkan video driver on an Intel HD Graphics 610 (Intel CPU)
I've also compiled the 1.7.6 version of whisper.cpp without vulkan support, and that whisper.cpp version is running flawless.
These are the switches I used for cmake (with vulkan support): cmake -B build -DGGML_VULKAN=1 -DWHISPER_FFMPEG=yes -DGGML_BLAS=1 -DWHISPER_SDL2=ON
I have also compiled a version with only -DGGML_VULKAN=1 enabled, but with the same results (crash and errors).
When I try to transcribe an audio file, the whisper-cli app is running for a while and produces output (sometimes for a few minutes), and then it suddenly crashes with the following error message:
---[CUT]--- [New LWP 11106] [New LWP 11113] [New LWP 11114] [New LWP 11115] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". 0x00007f4df20f1c17 in __GI___wait4 (pid=11231, stat_loc=0x0, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30 30 ../sysdeps/unix/sysv/linux/wait4.c: No such file or directory. #0 0x00007f4df20f1c17 in __GI___wait4 (pid=11231, stat_loc=0x0, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30 #1 0x00007f4df095021b in ggml_print_backtrace () at ./whisper.cpp-1.7.6/whisper/lib/libggml-base.so #2 0x00007f4df095e599 in ggml_uncaught_exception() () at ./whisper.cpp-1.7.6/whisper/lib/libggml-base.so #3 0x00007f4df06a8e1a in () at /lib/x86_64-linux-gnu/libstdc++.so.6 #4 0x00007f4df06a8e85 in () at /lib/x86_64-linux-gnu/libstdc++.so.6 #5 0x00007f4df06a90d8 in () at /lib/x86_64-linux-gnu/libstdc++.so.6 #6 0x00007f4de9a2978f in vk::(anonymous namespace)::throwResultException(vk::Result, char const*) () at ./whisper.cpp-1.7.6/whisper/lib/libggml-vulkan.so #7 0x00007f4de9a2b96e in ggml_vk_submit(std::shared_ptr<vk_context_struct>&, vk::Fence) [clone .cold] () at ./whisper.cpp-1.7.6/whisper/lib/libggml-vulkan.so #8 0x00007f4de9ac8e20 in ggml_vk_buffer_read(std::shared_ptr<vk_buffer_struct>&, unsigned long, void*, unsigned long) () at ./whisper.cpp-1.7.6/whisper/lib/libggml-vulkan.so #9 0x00007f4de9ac92b7 in ggml_backend_vk_buffer_get_tensor(ggml_backend_buffer*, ggml_tensor const*, void*, unsigned long, unsigned long) () at ./whisper.cpp-1.7.6/whisper/lib/libggml-vulkan.so #10 0x00007f4df26ab3d3 in whisper_decode_internal(whisper_context&, whisper_state&, whisper_batch const&, int, bool, bool ()(void), void*) () at ./whisper.cpp-1.7.6/whisper/lib/libwhisper.so.1 #11 0x00007f4df26c1201 in whisper_full_with_state () at ./whisper.cpp-1.7.6/whisper/lib/libwhisper.so.1 #12 0x00007f4df26c5a5c in whisper_full () at ./whisper.cpp-1.7.6/whisper/lib/libwhisper.so.1 #13 0x0000557913258fe4 in main () [Inferior 1 (process 11102) detached] terminate called after throwing an instance of 'vk::DeviceLostError' what(): vk::Queue::submit: ErrorDeviceLost ---[CUT]---
And sometimes, the transcription process does not output any recognized text, but only exclamation marks, like this: [00:00:00,580 --> 00:00:04,530]: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Again, this only happens when using a whisper.cpp version with vulkan support enabled. Never had such an output with a version without vulkan.
I have also tried to compile the latest whisper.cpp version 1.8.0, with vulkan support, but with that version even the compilation process breaks with errors: (because of vulkan) These are unfortunately just a few error lines, because there came thousands of them during compilation... And as far as I could see they were all ggml-vulkan.cpp related.
---[CUT]--- whisper.cpp-1.8.0/ggml/src/ggml-vulkan/ggml-vulkan.cpp:1761:84: error: invalid initialization of reference of type ‘const vk::DispatchLoaderDynamic&’ from expression of type ‘vk::detail::DispatchLoaderDynamic’ ... whisper.cpp-1.8.0/ggml/src/ggml-vulkan/ggml-vulkan.cpp:1780:93: error: could not convert ‘ggml_vk_default_dispatcher()’ from ‘vk::detail::DispatchLoaderDynamic’ to ‘const vk::DispatchLoaderDynamic&’ 1780 | dBuffer> cmd_buffers = device->device.allocateCommandBuffers(command_buffer_alloc_info); | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~ | | | vk::detail::DispatchLoaderDynamic ---[CUT]---
