CommandBufferGL::drawElements Crash

[kiranb47]

• axmol version: Latest dev branch
• devices test on:
• developing environments
  • NDK version: r19c
  • Xcode version: 12.4
  • Visual Studio:
    • VS version: 2019 (16.11), 2022 (17.4)
    • MSVC version: 1929, 1934
    • Windows SDK version: 10.0.22621.0
  • cmake version: Steps to Reproduce:

We have migrated our Match Animal game to Axmol from Cocos2dx V3 latest yesterday. Getting this crash now. How to fix this issue?

backtrace: #00 pc 0x00000000005ffa58 /data/app/games.spearmint.matchanimal-ayOwq5oTYpeKYIRIPMWVUQ==/lib/arm64/libMatchAnimal.so (ax::Renderer::TriangleCommandBufferManager::createBuffer()+1681) #01 pc 0x00000000005fdbc0 /data/app/games.spearmint.matchanimal-ayOwq5oTYpeKYIRIPMWVUQ==/lib/arm64/libMatchAnimal.so (ax::Renderer::init()+964) #02 pc 0x000000000063b060 /data/app/games.spearmint.matchanimal-ayOwq5oTYpeKYIRIPMWVUQ==/lib/arm64/libMatchAnimal.so (ax::Director::setOpenGLView(ax::GLView*)+394) #03 pc 0x00000000005640cc /data/app/games.spearmint.matchanimal-ayOwq5oTYpeKYIRIPMWVUQ==/lib/arm64/libMatchAnimal.so (Java_org_axmol_lib_AxmolRenderer_nativeInit+99) #04 pc 0x00000000001c5ff0 /data/app/games.spearmint.matchanimal-ayOwq5oTYpeKYIRIPMWVUQ==/oat/arm64/base.odex

[rh101]

How to fix this issue?

At risk of stating the obvious, have you actually attempted to use a debugger or other methods of tracking down why it crashes? A breakpoint in the right place may be the only thing you need to find out the issue. You've provided no source code, no context to the crash, whether it's on a real device or emulator, or what the app was doing at the time.

It's something in your own code causing this issue, and unlikely given to be something in the game engine, and given that you ported it from what I can only assume is a working version of the app, then something like this must be trivial for you to fix.

[kiranb47]

Hey @rh101 The issue is not reproducible in any of our test devices. Crash logs are from Google Play live users. 3.6% of our GPLAY users have the issue.

[kiranb47]

Another similar crash:

[split_config.armeabi_v7a.apk!libMatchAnimal.so] CommandBufferGL.cpp - ax::backend::CommandBufferGL::drawArrays(ax::backend::PrimitiveType, unsigned int, unsigned int, bool) SIGSEGV filter_list

date_range Last 28 days arrow_drop_down Affected users 9 Last 28 days Events 20 Last 28 days Last occurred 3 hours ago Last updated Today, 10:30 AM Affected users Apr 27, 2023 - May 25, 2023 Affected users Events By app version 10205 (1.2.5) By Android version Android 11 (SDK 30) Android 12 (SDK 31) By device samsung a10s samsung a02 Redmi ice TCL Cruze_Lite By issue visibility help_outline Foreground Stack trace help_outline 1 Sample attributes: samsung a10s (Galaxy A10s)

Android 11 (SDK 30)

Version: 10205 (1.2.5)Occurred: 3 hours ago
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 10195 >>> games.spearmint.matchanimal <<<

backtrace:
  #00  pc 0x00000000000701ae  /vendor/lib/egl/libGLESv2_mtk.so
  #01  pc 0x0000000000089ca1  /vendor/lib/egl/libGLESv2_mtk.so
  #02  pc 0x00000000000888d9  /vendor/lib/egl/libGLESv2_mtk.so (glDrawArrays+2592)
  #03  pc 0x000000000002d549  /vendor/lib/egl/libGLES_meow.so (MEOW::meow_call_ddk_gl_2_glDrawArrays(unsigned int, int, int)+20)
  #04  pc 0x00000000004255d7  /data/app/~~ZtkX28x-dzXdBYditxORXg==/games.spearmint.matchanimal-6vE2kmMTB0DCfes9iuUpeg==/split_config.armeabi_v7a.apk!libMatchAnimal.so (ax::backend::CommandBufferGL::drawArrays(ax::backend::PrimitiveType, unsigned int, unsigned int, bool)+222)
  #05  pc 0x000000000040feef  /data/app/~~ZtkX28x-dzXdBYditxORXg==/games.spearmint.matchanimal-6vE2kmMTB0DCfes9iuUpeg==/split_config.armeabi_v7a.apk!libMatchAnimal.so (ax::Renderer::drawCustomCommand(ax::RenderCommand*)+758)
  #06  pc 0x000000000040ff3d  /data/app/~~ZtkX28x-dzXdBYditxORXg==/games.spearmint.matchanimal-6vE2kmMTB0DCfes9iuUpeg==/split_config.armeabi_v7a.apk!libMatchAnimal.so (ax::Renderer::doVisitRenderQueue(std::__ndk1::vector<ax::RenderCommand*, std::__ndk1::allocator<ax::RenderCommand*>> const&)+398)
  #07  pc 0x000000000040f7d5  /data/app/~~ZtkX28x-dzXdBYditxORXg==/games.spearmint.matchanimal-6vE2kmMTB0DCfes9iuUpeg==/split_config.armeabi_v7a.apk!libMatchAnimal.so (ax::Renderer::visitRenderQueue(ax::RenderQueue&)+386)
  #08  pc 0x000000000040f539  /data/app/~~ZtkX28x-dzXdBYditxORXg==/games.spearmint.matchanimal-6vE2kmMTB0DCfes9iuUpeg==/split_config.armeabi_v7a.apk!libMatchAnimal.so (ax::Renderer::processGroupCommand(ax::GroupCommand*)+292)
  #09  pc 0x000000000040f8e3  /data/app/~~ZtkX28x-dzXdBYditxORXg==/games.spearmint.matchanimal-6vE2kmMTB0DCfes9iuUpeg==/split_config.armeabi_v7a.apk!libMatchAnimal.so (ax::Renderer::processRenderCommand(ax::RenderCommand*)+342)
  #10  pc 0x000000000040ff3d  /data/app/~~ZtkX28x-dzXdBYditxORXg==/games.spearmint.matchanimal-6vE2kmMTB0DCfes9iuUpeg==/split_config.armeabi_v7a.apk!libMatchAnimal.so (ax::Renderer::doVisitRenderQueue(std::__ndk1::vector<ax::RenderCommand*, std::__ndk1::allocator<ax::RenderCommand*>> const&)+398)
  #11  pc 0x000000000040f7d5  /data/app/~~ZtkX28x-dzXdBYditxORXg==/games.spearmint.matchanimal-6vE2kmMTB0DCfes9iuUpeg==/split_config.armeabi_v7a.apk!libMatchAnimal.so (ax::Renderer::visitRenderQueue(ax::RenderQueue&)+386)
  #12  pc 0x000000000040ff8b  /data/app/~~ZtkX28x-dzXdBYditxORXg==/games.spearmint.matchanimal-6vE2kmMTB0DCfes9iuUpeg==/split_config.armeabi_v7a.apk!libMatchAnimal.so (ax::Renderer::render()+415)
  #13  pc 0x00000000003d64b1  /data/app/~~ZtkX28x-dzXdBYditxORXg==/games.spearmint.matchanimal-6vE2kmMTB0DCfes9iuUpeg==/split_config.armeabi_v7a.apk!libMatchAnimal.so (ax::Scene::render(ax::Renderer*, ax::Mat4 const&, ax::Mat4 const*)+225)
  #14  pc 0x000000000043953d  /data/app/~~ZtkX28x-dzXdBYditxORXg==/games.spearmint.matchanimal-6vE2kmMTB0DCfes9iuUpeg==/split_config.armeabi_v7a.apk!libMatchAnimal.so (ax::Director::drawScene()+295)
  #15  pc 0x000000000043af85  /data/app/~~ZtkX28x-dzXdBYditxORXg==/games.spearmint.matchanimal-6vE2kmMTB0DCfes9iuUpeg==/split_config.armeabi_v7a.apk!libMatchAnimal.so (ax::Director::mainLoop()+1485)
  #16  pc 0x00000000000ad5d7  /data/app/~~ZtkX28x-dzXdBYditxORXg==/games.spearmint.matchanimal-6vE2kmMTB0DCfes9iuUpeg==/oat/arm/base.odex (art_jni_trampolin

[rh101]

Crash logs are from Google Play live users. 3.6% of our GPLAY users have the issue.

Is it limited to specific devices or Android versions, and if so, what devices and Android versions are having this issue?

I notice the armeabi_v7a in that dump. Is the crash limited to 32bit devices running armeabi_v7a?

Also, please put code tags around the on the crash dumps so they're formatted and easier to read. Just add 3 backtick ` symbols before and after the crash dumps.

[kiranb47]

@rh101 Issue also in arm64_v8 devices.

[split_config.arm64_v8a.apk!libMatchAnimal.so] CommandBufferGL.cpp - ax::backend::CommandBufferGL::drawElements(ax::backend::PrimitiveType, ax::backend::IndexFormat, unsigned long, unsigned long, bool)

backtrace:
  #00  pc 0x0000000000088b78  /apex/com.android.runtime/lib64/bionic/libc.so (__memcpy+232)
  #01  pc 0x00000000001eb86c  /vendor/lib64/egl/libGLESv2_adreno.so (!!!0000!6642a709aefb598e3c7ff817eaf381!03dd3ba!+3524)
  #02  pc 0x0000000000360b08  /vendor/lib64/egl/libGLESv2_adreno.so (!!!0000!928ddd272828b5ecd302b519177d41!03dd3ba!+904)
  #03  pc 0x0000000000332b54  /vendor/lib64/egl/libGLESv2_adreno.so (!!!0000!4c38aea95be9faf0b3861d1af73f50!03dd3ba!+2916)
  #04  pc 0x0000000000320078  /vendor/lib64/egl/libGLESv2_adreno.so (!!!0000!591ab8a9d75351b2e63b06236cc5c1!03dd3ba!+16)
  #05  pc 0x0000000000133bfc  /vendor/lib64/egl/libGLESv2_adreno.so (!!!0000!6b200851123c7898055fe62ff9f71f!03dd3ba!+1876)
  #06  pc 0x000000000012be84  /vendor/lib64/egl/libGLESv2_adreno.so (!!!0000!77df12deb6a622478efa8fb9929034!03dd3ba!+1004)
  #07  pc 0x000000000061f8a8  /data/app/~~jlm2IhawQGELdTRWJwWbqw==/games.spearmint.matchanimal-oUvgtVnie2xt30tW0BFwmg==/split_config.arm64_v8a.apk!libMatchAnimal.so (ax::backend::CommandBufferGL::drawElements(ax::backend::PrimitiveType, ax::backend::IndexFormat, unsigned long, unsigned long, bool)+242)
  #08  pc 0x00000000005fee60  /data/app/~~jlm2IhawQGELdTRWJwWbqw==/games.spearmint.matchanimal-oUvgtVnie2xt30tW0BFwmg==/split_config.arm64_v8a.apk!libMatchAnimal.so (ax::Renderer::drawCustomCommand(ax::RenderCommand*)+752)
  #09  pc 0x00000000005fef04  /data/app/~~jlm2IhawQGELdTRWJwWbqw==/games.spearmint.matchanimal-oUvgtVnie2xt30tW0BFwmg==/split_config.arm64_v8a.apk!libMatchAnimal.so (ax::Renderer::doVisitRenderQueue(std::__ndk1::vector<ax::RenderCommand*, std::__ndk1::allocator<ax::RenderCommand*>> const&)+398)
  #10  pc 0x00000000005fe4b0  /data/app/~~jlm2IhawQGELdTRWJwWbqw==/games.spearmint.matchanimal-oUvgtVnie2xt30tW0BFwmg==/split_config.arm64_v8a.apk!libMatchAnimal.so (ax::Renderer::visitRenderQueue(ax::RenderQueue&)+386)
  #11  pc 0x00000000005fe27c  /data/app/~~jlm2IhawQGELdTRWJwWbqw==/games.spearmint.matchanimal-oUvgtVnie2xt30tW0BFwmg==/split_config.arm64_v8a.apk!libMatchAnimal.so (ax::Renderer::processGroupCommand(ax::GroupCommand*)+292)
  #12  pc 0x00000000005fe644  /data/app/~~jlm2IhawQGELdTRWJwWbqw==/games.spearmint.matchanimal-oUvgtVnie2xt30tW0BFwmg==/split_config.arm64_v8a.apk!libMatchAnimal.so (ax::Renderer::processRenderCommand(ax::RenderCommand*)+342)
  #13  pc 0x00000000005fef04  /data/app/~~jlm2IhawQGELdTRWJwWbqw==/games.spearmint.matchanimal-oUvgtVnie2xt30tW0BFwmg==/split_config.arm64_v8a.apk!libMatchAnimal.so (ax::Renderer::doVisitRenderQueue(std::__ndk1::vector<ax::RenderCommand*, std::__ndk1::allocator<ax::RenderCommand*>> const&)+398)
  #14  pc 0x00000000005fe4b0  /data/app/~~jlm2IhawQGELdTRWJwWbqw==/games.spearmint.matchanimal-oUvgtVnie2xt30tW0BFwmg==/split_config.arm64_v8a.apk!libMatchAnimal.so (ax::Renderer::visitRenderQueue(ax::RenderQueue&)+386)
  #15  pc 0x00000000005fef74  /data/app/~~jlm2IhawQGELdTRWJwWbqw==/games.spearmint.matchanimal-oUvgtVnie2xt30tW0BFwmg==/split_config.arm64_v8a.apk!libMatchAnimal.so (ax::Renderer::render()+415)
  #16  pc 0x00000000005b17b8  /data/app/~~jlm2IhawQGELdTRWJwWbqw==/games.spearmint.matchanimal-oUvgtVnie2xt30tW0BFwmg==/split_config.arm64_v8a.apk!libMatchAnimal.so (ax::Scene::render(ax::Renderer*, ax::Mat4 const&, ax::Mat4 const*)+225)
  #17  pc 0x000000000063a760  /data/app/~~jlm2IhawQGELdTRWJwWbqw==/games.spearmint.matchanimal-oUvgtVnie2xt30tW0BFwmg==/split_config.arm64_v8a.apk!libMatchAnimal.so (ax::Director::drawScene()+295)
  #18  pc 0x000000000063ca68  /data/app/~~jlm2IhawQGELdTRWJwWbqw==/games.spearmint.matchanimal-oUvgtVnie2xt30tW0BFwmg==/split_config.arm64_v8a.apk!libMatchAnimal.so (ax::Director::mainLoop()+1485)
  #19  pc 0x00000000000ada90  /data/app/~~jlm2IhawQGELdTRWJwWbqw==/games.spearmint.matchanimal-oUvgtVnie2xt30tW0BFwmg==/oat/arm64/base.odex (art_jni_trampoline+144)
  #20  pc 0x0000000002018c7c  /memfd:jit-cache (org.axmol.lib.AxmolRenderer.onDrawFrame+220)
  #21  pc 0x000000000069f968  /system/framework/arm64/boot-framework.oat (android.opengl.GLSurfaceView$GLThread.guardedRun+3992)
  #22  pc 0x00000000006a01b0  /system/framework/arm64/boot-framework.oat (android.opengl.GLSurfaceView$GLThread.run+224)
  #23  pc 0x0000000000134564  /apex/com.android.art/lib64/libart.so (art_quick_invoke_stub+548)
  #24  pc 0x0000000000198e94  /apex/com.android.art/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+204)
  #25  pc 0x0000000000532198  /apex/com.android.art/lib64/libart.so (art::(anonymous namespace)::InvokeWithArgArray(art::ScopedObjectAccessAlreadyRunnable const&, art::ArtMethod*, art::(anonymous namespace)::ArgArray*, art::JValue*, char const*)+104)
  #26  pc 0x0000000000533398  /apex/com.android.art/lib64/libart.so (art::JValue art::InvokeVirtualOrInterfaceWithJValues<art::ArtMethod*>(art::ScopedObjectAccessAlreadyRunnable const&, _jobject*, art::ArtMethod*, jvalue const*)+440)
  #27  pc 0x00000000005808b8  /apex/com.android.art/lib64/libart.so (art::Thread::CreateCallback(void*)+1272)
  #28  pc 0x00000000000f40c4  /apex/com.android.runtime/lib64/bionic/libc.so (__pthread_start(void*)+64)
  #29  pc 0x000000000008ed10  /apex/com.android.runtime/lib64/bionic/libc.so (__start_thread+64)

[rh101]

@kiranb47 Crashes that affect a small subset of users may have very specific triggers, and since you are the one with access to the source code, you're in the best position to narrow down the possible causes. Do you collect any form of diagnostic information from your application? Can you tell what a user is doing in the app just prior to the crash?

[solan-solan]

According to provided logs, all your issues are related to gl buffers. The falt occurred inside gldrawelements, gldrawarrays and while buffer creating. For example, axmol supports int as base type for index array as opposed to cocos. I would check how your buffers creating in your code at first, since the issues looks logical. You created them manually?

[kiranb47]

Hey @rh101 Firebase is not reporting this issue, so not able to identify pre crash events logs. Hey @solan-solan I have no technical knowledge about OpenGL. We have not applied any changes to axmol source code for this game.

[kiranb47]

@rh101 @solan-solan Issue is related to UI ScrollView

[rh101]

It's great that you have managed to narrow down the issue. Can you show the snippet of code related to how you use the ui::ScrollView? Include how you set it up/initialize it, and any other related code. If you could create a small project with just the ui::ScrollView usage then that would help.

[crazyhappygame]

FYI. I migrated my app from cocos2d-x 3.17 to latest axmol (4e664e6681457d653c88e072c0ef30bcceac77e6) and see in Google Play similar crashes in drawElements and drawArrays

ax::backend::CommandBufferGL::drawElements(ax::backend::PrimitiveType, ax::backend::IndexFormat, unsigned long, unsigned long, bool)

and

ax::backend::CommandBufferGL::drawArrays(ax::backend::PrimitiveType, unsigned long, unsigned long, b

In my game I use ui::ScrollView as well. I used many (100-1000) Sprite and Scale9Sprite. @kiranb47 how did you narrow problem to ui::ScrollView ?

App with cocos2d-x 3.17 has ~0 crashes after upgrading I see significant increase. I can not reproduce issue on local devices ...

Please check below screenshot how much crashes increased in Google Play after update:

[DelinWorks]

is your project open source? I can debug and potentially fix this for you if so. otherwise why not compile a small project that showcases the issue like @rh101 said?

[crazyhappygame]

@DelinWorks Thank you for an offer.

The problem is that I can not reproduce problem locally ... I see crashes only in Google Play. I am so sorry my project is not open source. Based on above callstacks it looks like we crash somewhere here

void CommandBufferGL::drawElements(PrimitiveType primitiveType,
                                   IndexFormat indexType,
                                   std::size_t count,
                                   std::size_t offset,
                                   bool wireframe)
{
    prepareDrawing();
#ifndef AX_USE_GLES  // glPolygonMode is only supported in Desktop OpenGL
    if (wireframe) glPolygonMode(GL_FRONT_AND_BACK, GL_LINE);
#else
    if (wireframe) primitiveType = PrimitiveType::LINE;
#endif
    glBindBuffer(GL_ELEMENT_ARRAY_BUFFER, _indexBuffer->getHandler());
    glDrawElements(UtilsGL::toGLPrimitiveType(primitiveType), count, UtilsGL::toGLIndexType(indexType),
                   (GLvoid*)offset);

That could mean that _indexBuffer is incorrect or parameters to glDrawElements are incorrect.

Unfortunately I am not familiar with GL. Do you think that it would be possible having above callstack analyze existing code and narrow problem through code inspection? @DelinWorks

[DelinWorks]

You were able to produce it on your device or is that a google analysis ghost crash

[DelinWorks]

if google analysis could show you which population of devices it crashes on it would be helpful

[crazyhappygame]

@DelinWorks I can not reproduce crash on my device. I see crash only in Google Play console. In Google Play console I see crash on following devices: OPPO OP56F5 (A17) Android 12 (SDK 31) Redmi lemon (Redmi 9T) Android 10 (SDK 29) samsung a70q (Galaxy A70) Android 11 (SDK 30) samsung starlte (Galaxy S9) Android 10 (SDK 29)

FYI. This game is was updated 3 days ago and only ~2K users updated version.

[DelinWorks]

unfortunately you'll need to get your hands on one of those devices to test it, or try to roughly match the specs, os version, and sdk of the devices to hopefully regenerate the bug.

[rh101]

@crazyhappygame @kiranb47

Is there any chance you could create a debug version of your applications, and upload them to either the "Internal testing" track in your Google Play Console? I can provide you with a gmail address to add to the internal tester list, and I suggest others here provide an email address too if they can help (best to generate a new gmail address for this purpose, and not use your personal address). Once you upload the APK to that internal testing track, you can share the link to it so that the users on your testers list may download it.

This issue has me a little concerned, and I'm going to try to get my hands on at least one of the specific devices that seem to be having this issue. Also, describe exactly what it is that needs to be done in the application to cause it to crash (like which screen to go to, what exactly to click on etc).

Also, @crazyhappygame which of the devices that you listed have the most crashes?

Another thing, we really need a sample of the ui::ScrollView usage. We don't need an entire app source code, just any code related to how the ui::ScrollView is being created and used. If you can extract that section of code from your apps, remove/rename any private/proprietary info, and provide that, then it may help others replicate this issue.

[solan-solan]

@crazyhappygame

I used many (100-1000) Sprite and Scale9Sprite. @kiranb47 how did you narrow problem to ui::ScrollView ?

May be this is some type of memory leack? Do you sure that gl buffers and internal arrays of your sprites properly free? And is there opportunity to analysis how many time user spent in game before crashing?

[crazyhappygame]

@solan-solan I used just Sprite and Scale9Sprite with image. No custom tricks, shaders etc. Not sure how to get information how much time user spend in the app before crash....

What I have seen in stacktrace that it crashes in memcpy. For me that means that there is no problem with allocation but with access to unavailable memory.

@rh101 @DelinWorks @kiranb47 do you know how to build entire (all 3rdparty libs, axmol, gsme) axmol with address sanitizer? https://developer.android.com/ndk/guides/asan#cmake ? I have strong feeling that it can solve this mystery.

[crazyhappygame]

@rh101 could you send me instructions how to build debug version of app and Gmail address? (Gradle assembleDebug)?

[rh101]

@rh101 could you send me instructions how to build debug version of app and Gmail address? (Gradle assembleDebug)?

If you're using Android Studio, then just use the menus to configure it:

and this window pops up:

There may be a way to do it from the command line too, but I can't recall it at the moment, so perhaps someone else knows how to do it.

EDIT: email removed. @crazyhappygame if you need it again, I'll post it up.

[rh101]

@rh101 @DelinWorks @kiranb47 do you know how to build entire (all 3rdparty libs, axmol, gsme) axmol with address sanitizer? https://developer.android.com/ndk/guides/asan#cmake

Are the instructions on that page not working for you?

[crazyhappygame]

Below have to be set for all targets including all 3rdparty libs I do not know how to rebuild all android deps

set_target_properties(${TARGET} PROPERTIES LINK_FLAGS -fsanitize=address)```

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[rarepixels]

Hi all, I also recently migrated my app, that went from cocos2d trough all cocos2d-x versions until v4, and had virtually no crashes, and afther the last migration to axmol release 1.0.0, I also saw a huge increase in android in ax::backend::CommandBufferGL::setUniforms.

As kiranb47 commented, I also never saw the crash on any live test or during development.

My crashlytics reports, gives a little more information, maybe this can help.

[rh101]

@rarepixels Would you happen to have more info about the devices that crash? For instance, is it limited to specific devices, and if so, which ones? Does it happen at any time after the app starts, or after a specific period of time? Any other info you can provide would be helpful.

[sarooptech]

• Try compressing the PNG files with TinyPNG and also replace the WEBP image with compressed PNG. Maybe that will solve your Crash problem.
• If still have that crash problem try running the game on an iOS simulator. Because caught some bugs that did not occur in my Android devices but I was able to reproduce them inside iOS simulators and iPhone Devices.

[halx99]

Maybe caused by context lost, sometimes go back to game from sdk Activity

[sarooptech]

It could be. Android does have so many unpredictable behaviors.