Landscape scanning issues

[Eric-Kosovec]

For the project I'm working on, we are using your quagga2-reader-qr project as a base for integrating the QR library we're using (Nimiq qr-scanner) into Quagga. We were having issues with landscape scanning specifically only with QR codes.

I noticed orientation changes brought up maybe a few times and you mentioned you were aware of a possible issue. The issue was the inputImageWrapper: ImageWrapper given to decodeImage in the QRCodeReader class, the width and height always stay the dimensions of portrait mode instead of swapping when turned to landscape.

This was causing the input image to be warped and thus not be able to be scanned - at least not by the library we're using. Most of the time, I would have to hold the phone at an extreme angle to the QR code to get it to scan. I'm thinking this is because Quagga only grabs the video element dimensions once, so it never gets the updated landscape dimensions.

These are screenshots of the ImageWrapper from Quagga drawn onto a canvas built from the wrapper's dimensions. First one portrait and second landscape.

[github-actions[bot]]

Thank you for filing an issue! Please be patient. :-)

[ericblade]

Thanks for the analysis! I'm trying to wrap my mind around the implications here. I wonder if catching orientationchange or resize events and reconfiguring some spot would help.

I've been doing professional work for a good amount of time right now, and haven't had much time to work in Quagga or my other projects... so I'm not able to just rip off a good idea to check here.. thinking about it, though.

I wonder if a workaround would be to catch orientationchange and then stop() and start() again. As is, an orientation change inside an app can be computationally expensive and slow with the camera open, but I wonder if that would at least work around the problem.

As to actually fixing the problem.. that's going to take some poking around and figuring out what needs to be reconfigured and how.

[Eric-Kosovec]

One workaround I found was to just call init(), but then the video kind of disappears for a second and sometimes goes pink, so had to just pop up a loading icon and hide the video until the element stabilized, which is very much not user-friendly when they change the orientation. I think the same thing happened for stop() and start(), though I actually can't remember if that combo fixed it. I want to say it didn't fix it unless I also called init().

Ultimately, I ended up not using quagga2-reader-qr. I have Quagga doing the live scanning, while I have an interval running to grab frames from the video element and then feed them into the QR scanner. For now this works, but I would eventually like to restore the code back to using quagga2-reader-qr because it's a lot more succinct.

I am still relatively new to JavaScript and web dev stuff, but how I created the canvas for copying the frames to was getting the video element's dimensions, which seemed to change when the orientation does, so there was no need to listen for orientation changes. I don't know if it's entirely reliable though, but seems to work fine. I'm thinking the issue is somewhere in here and/or the frame_grabber_browser. I couldn't say where exactly other than probably one of those sizes it gets is not the proper one and is using old data or maybe something else is wrong with the data processing it's doing.

[ericblade]

OK, so are you saying that this does affect the image that is given out to the external reader, but does not affect the image for the built-in readers? Or do you not know the answer to if the built-in readers are affected? If only external readers are affected then it's a different nature of problem than if it affects everything.

[Eric-Kosovec]

I'm not sure, but if the built-in readers get their ImageWrapper from the same place, I would imagine they are affected too.

[ericblade]

yep, that's what i'm not sure of. I honestly don't know when I'm going to get time to examine this, we've had a pretty tragic last several weeks around us here, but I really appreciate how far you've gone to illustrate the problem. I'm pretty confident with that information, it shouldn't be too hard to come up with an answer once i have time to sit down and wrap my head around the code again.

[ericblade]

video element's dimensions, which seemed to change when the orientation does

realizing right now, that that might be an issue. It might be possible to make quagga more intelligent as to it's input image dimensions, but that's something that'd be more in line with the major restructuring i'm doing on the 2.0 branch, than something i'd wa nt to add as a feature in the current master line. Having Quagga pay attention to the input image size (perhaps via the dimensions of it's video element when it's using live camera, or the actual image resolution when using input images) and adjusting as necessary on an element resize would probably be valuable, but could be a fair undertaking.

Right now, i'm messing with fixing some issues seen in my in-use app when switching between portrait and landscape modes. Previously, I had a fixed size video/canvas/etc that actually functioned fine, but aesthetically it was a mess. Also, the canvas didn't (and still doesn't) line up correctly when drawing boxes, after switching modes -- and that's a fault of Quagga not knowing what do do, I think.

Here's a snip of some of my React component, not all context is included here, but it should demonstrate what I'm trying to do

    const deviceOrientation = useDeviceOrientation();
    const isLandscape = deviceOrientation.alpha === 90 || deviceOrientation.alpha === 270;
    const actualWidth = isLandscape ? 480 : window.innerWidth;
    const actualHeight = isLandscape ? window.innerWidth : 480;

    return (
        <div className={css.scannerContainer} ref={scannerRef}>
            {/* // TODO: these constants are screwing up rendering when larger than 640x480. */}
            <video style={{ width: actualWidth , height: actualHeight }} />
            <canvas className="drawingBuffer" width={actualWidth} height={actualHeight} />
            {
                scannerOpen && (
                    <Scanner
                        cameraId={cameraId}
                        scannerRef={scannerRef}
                        onScannerReady={onScannerReady}
                        onDetected={onDetected}
                    />
                )
            }
            <button className={css.torchButton} onClick={onTorchClick}>
                <img className={css.torchImage} src={bTorchOn ? torchOn : torchOff} alt="torch" />
            </button>
        </div>
    );

[ericblade]

so... this code handles resizing the video/canvas element into sane and knowable dimensions whenever the device rotates. It needs some more work done on it, because I probably want a much smaller height on the whole thing when you're in landscape mode, but flipping and maintaining the correct aspect ratio is what i was trying to achieve here.

Probably after I get it visually working correctly in my own app with the relatively correct sizes in both portrait and landscape, i will investigate adding a way to update quagga manually, and then investigate if it is possible to detect it automatically at the quagga level, without also interfering with anything (or at least making it an option to turn on, i guess)

Unfortunately, I can't say when I will get to doing this. This is the first weekend I've had in months to concentrate on my own app, which also required some additional work inside quagga, so I was able to do some fixes to some things today. A deep dive into anything isn't something I see happening perhaps until the holiday break.

[Eric-Kosovec]

Awesome, I appreciate the update. So far my temporary fix seems to be holding up, so we can probably wait until whenever you get to it.