[Feature request] OpenGL example usages ByteBuffer interpretations

[emitchel]

What's your feature request? Please describe. This sample set does a great job at showing us how to overlay graphics in the View hierarchy, using bitmaps and the canvas. Although, generating bitmaps is intensive, and when considering a video capturing scenario, we're asking a lot of the system to generate new bitmaps for every frame. It would be helpful to see how we can leverage the ByteBuffers in the context of OpenGL shaders. e.g. SegmentationMask byte buffer used to change the background of the camera feed, per texture coordinate. This offloads much of the heavy lifting to the GPU.

Have we seen examples of this? I've been trying to interpret the byte buffer as an unsigned byte then setting the TextImage2d to represent luminance, but it seems the ordering of the ByteBuffer or dimensions seems to screw up the Fragment shader, as there's duplication of artifacts and it doesn't map well.

Mobile environment Android

Additional context I'm not a OpenGL ES expert, still learning! Here's an isolated project to show how I'm setting it up - I'm pretty sure I'm abusing the texture setup, https://github.com/emitchel/ElliotOpenGLPlayground/blob/master/app/src/main/java/com/opengl/camera/programs/SegmentationOnlyCameraProgram.kt

Update

It turns out OpenGL expects unsigned bytes to work with Luminance in the following fashion, and the mask bytebuffer returned from SegmentationMask is signed (-127, 126).

GLES20.glTexSubImage2D(
                GLES20.GL_TEXTURE_2D,
                0,
                0,
                0,
                byteBufferWidth,
                byteBufferHeight,
                GLES20.GL_LUMINANCE,
                GLES20.GL_UNSIGNED_BYTE,
                byteBuffer!!
            )

So I can convert it in the following fashion, and that does allow it to be rendered in the fragment shader, but it's less performant than building a bitmap of the original image and bitmap overlay... which makes me think I'm going about it wrong entirely. Any help would be greatly appreciated!

    val mask = segmentationMask.getBuffer()
    val maskWidth = segmentationMask.getWidth()
    val maskHeight = segmentationMask.getHeight()

    val byteBufferSize = maskWidth * maskHeight // one byte per pixel for LUMINANCE
    val resultBuffer = ByteBuffer.allocateDirect(byteBufferSize)

    for (y in 0 until maskHeight) {
        for (x in 0 until maskWidth) {
            val foregroundConfidence = mask.getFloat()
            val byteValue = (foregroundConfidence * 255.0f).toInt().toByte()
            resultBuffer.put(byteValue)
        }
    }

    resultBuffer.rewind()
    return resultBuffer
}

[emitchel]

enablingRawMask helps drastically, which makes sense and is probably the right approach here. Either way, still curious how other integrations are solving this!