Running two detectors at the same time

[benedikt90]

Hi!

I was wondering if it is possible to run two detectors, e.g. Text/Barcode at the same time? I'm new to Dart/Flutter, so this might be a basic question - sorry for that.

I started by changing the typedef HandleDetection<T> = Future<List<T>> Function(FirebaseVisionImage image);. However, I'm not sure if that's a good direction. I just want to be able to give a list of detectors, and then process these in order. I'm failing at fixing this error:

{
	"resource": "/C:/Users/dev/Documents/code/flutter_camera_ml_vision-master/lib/utils.dart",
	"owner": "dart",
	"code": "return_of_invalid_type",
	"severity": 8,
	"message": "The return type 'Future<List<T>>' isn't a 'Future<T>', as defined by the method '_detect'.",
	"startLineNumber": 43,
	"startColumn": 10,
	"endLineNumber": 43,
	"endColumn": 150,
}

I'm having issues understanding the definition of the _detect<T> function. So I'm not sure how to handle this..

Cheers!

[catalin-apostu]

Hi reijin90!

I had the same need a few days ago, to do barcode and text recognition on the same frame.

First I have tried to solve it by changing the _processImage method in flutter_camera_ml_vision.dart (line 314) to work with a list of detectors (like you did). The app crushed after few seconds with out of memory, when testing (on a Pixel 2 XL device). Do not know why (also did not spend too much time on investigating), but I assume that running to detectors on the same CameraImage is resource intensive and something went wrong handling the memory allocation.

Then I have tried to replace the detector in CameraMlVision<T> to take a barcode or text detector alternatively, by setting _currentDetector to DetectorType.text after detecting the barcode and other way around. I have also used CameraMlVision<dynamic> so that I could get both List<Barcode> and VisionText as results. This approach works good. Got inspired in the implementation by the original MLVision example where they also switch between detectors.

enum DetectorType {
  barcode,
  text,
}

final BarcodeDetector _barcodeDetector =
      FirebaseVision.instance.barcodeDetector();
final TextRecognizer _textRecognizer =
      FirebaseVision.instance.textRecognizer();

DetectorType _currentDetector = DetectorType.barcode;

 _setCurrentDetector(DetectorType detectorType) {
    setState(() {
      _currentDetector = detectorType;
    });
 }

CameraMlVision<dynamic>(
      key: _scanKey,
      detector: getDetectionMethod(_currentDetector),
      resolution: ResolutionPreset.high,
      onResult: mounted ? _onResult : null,
      onDispose: () {
          _barcodeDetector.close();
          _textRecognizer.close();
      },
),

Future<dynamic> Function(FirebaseVisionImage image) getDetectionMethod(
    DetectorType detectorType,
  ) {
    switch (detectorType) {
      case DetectorType.text:
        return _textRecognizer.processImage;
      case DetectorType.barcode:
        return _barcodeDetector.detectInImage;
      default:
        throw FallThroughError();
    }
}

_onResult(dynamic result) {
    if (result is VisionText) {
      // process recognized text
    } else if (result is List<Barcode>) {
      // process recognized barcode
    }
}

Hope this helps you for the moment.

Cheers!

[benedikt90]

@catalin-apostu thanks for taking the time to answer my question in such detail, I highly appreciate it! I'll look into it.

Cheers!