OverlayEffect sample code

[Tachi107]

Hi all, I'm looking into using the new OverlayEffect class in my app to draw bounding boxes and labels of objects recognized by ML Kit's implementation of ImageAnalysis.Analyzer (i.e. MlKitAnalyzer).

I'm currently trying to keep things as simple as possible - I'm using CameraController - but I don't understand how I should hook up an OverlayEffect to it. An example would be really helpful, but I currently cannot find any in the camera-samples repository.

For instance, how would you render bounding boxes in an app using ML Kit's default Object Recognition model? Here's a small sample app where you could (hopefully) add an OverlayEffect.

class MainActivity : AppCompatActivity() {
    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        setContentView(R.layout.activity_main)

        val previewView = findViewById<PreviewView>(R.id.previewView)
        val cameraController = LifecycleCameraController(baseContext)
        cameraController.bindToLifecycle(this)
        cameraController.cameraSelector = CameraSelector.DEFAULT_BACK_CAMERA
        previewView.controller = cameraController

        val objectDetector = ObjectDetection.getClient(
            ObjectDetectorOptions.Builder()
                .setDetectorMode(ObjectDetectorOptions.STREAM_MODE)
                .enableClassification()
                .build()
        )

        cameraController.setImageAnalysisAnalyzer(
            ContextCompat.getMainExecutor(this),
            MlKitAnalyzer(
                listOf(objectDetector),
                CameraController.COORDINATE_SYSTEM_VIEW_REFERENCED,
                ContextCompat.getMainExecutor(this)
            )
            { result: MlKitAnalyzer.Result? ->
                if (result != null) {
                    val values = result.getValue(objectDetector)
                    if (values != null) {
                        for (value in values) {
                            TODO("Render bounding box with OverlayEffect")
                            println(value.boundingBox.flattenToString())
                        }
                    }
                }
            }
        )
    }
}

Thanks!

[Tachi107]

Hi, I saw the introduction of the new COORDINATE_SYSTEM_SENSOR option which, according to the commit which introduced it (by @xizhang), should be useful to make use of the new OverlayEffect API. Still, no sample code is present in the repository. Where can we find some helpful example showing how to make use of this efficient API together with an ImageAnalyzer like ML Kit or MediaPipe?

Thanks!

[xizhang]

We haven't gotten around to create an official code sample yet. For the time being, you can checkout the following change.

SyncedOverlayEffect contains the code for syncing the ML Kit frame to the preview/video frame.

OverlayFragment shows how to use it with ML Kit. Instead of using -1 for the target, you can use the new COORDINATE_SYSTEM_SENSOR in its place.

Please feel free to tag me if you run into any issues.