On the Windows app SDK using CameraHelper, FrameArrived time execution for a period of time will not be triggered.

[GreenShadeZhang]

Describe the bug

When I use wasdk 1.1.2, the processing of camera frame data, at first the FrameArrived event is normal, but after a period of time, the event does not trigger, I suspect it is was wasdk bug, but I will confirm it here first. The uwp code is normal like this https://github.com/GreenShadeZhang/GreenShade.UWPDemo/tree/master/GreenShade.ML.EmoticonDetection

But when the code is ported to winui, it can only be normal for a while.

https://github.com/GreenShadeZhang/GreenShade.UWPDemo/tree/master/GreenShade.WinUI.EmoticonDetection

Regression

No response

Reproducible in sample app?

• [ ] This bug can be reproduced in the sample app.

Steps to reproduce

1. Download the code
2. Run the relevant code

Expected behavior

I want the FrameArrived event of the video stream to work fine on the Windows app SDK so I can do emoticon analysis.

Screenshots

No response

Windows Build Number

• [ ] Windows 10 1809 (Build 17763)
• [ ] Windows 10 1903 (Build 18362)
• [ ] Windows 10 1909 (Build 18363)
• [ ] Windows 10 2004 (Build 19041)
• [ ] Windows 10 20H2 (Build 19042)
• [ ] Windows 10 21H1 (Build 19043)
• [X] Windows 11 21H2 (Build 22000)
• [ ] Other (specify)

Other Windows Build number

No response

App minimum and target SDK version

• [ ] Windows 10, version 1809 (Build 17763)
• [ ] Windows 10, version 1903 (Build 18362)
• [ ] Windows 10, version 1909 (Build 18363)
• [ ] Windows 10, version 2004 (Build 19041)
• [ ] Other (specify)

Other SDK version

No response

Visual Studio Version

2022

Visual Studio Build Number

No response

Device form factor

Desktop

Nuget packages

No response

Additional context

No response

Help us help you

No.

[msftbot[bot]]

Hello GreenShadeZhang, thank you for opening an issue with us!

I have automatically added a "needs triage" label to help get things started. Our team will analyze and investigate the issue, and escalate it to the relevant team if possible. Other community members may also look into the issue and provide feedback 🙌

[michael-hawker]

@GreenShadeZhang I don't believe any of the Camera/Media stuff is enabled in the WindowsAppSDK yet (at least to the point it's exposed in the same way that we consume in the Toolkit). We've been waiting for them to enable support for these scenarios. I know that's definitely true for the CameraPreview which uses the MediaPlayerElement, but not sure what's supported or not 100% with the underlying MediaFrameReader api the CameraHelper uses.

I know in the docs here and here they're only calling out the CaptureCameraUI from the Windows.Media.Capture namespace, so not sure if this is a missing gap in documentation or their radar/planning.

I'd recommend opening a discussion on the Windows App SDK repo to be sure of what is and isn't supposed to be working at this time.

[GreenShadeZhang]

@GreenShadeZhang I don't believe any of the Camera/Media stuff is enabled in the WindowsAppSDK yet (at least to the point it's exposed in the same way that we consume in the Toolkit). We've been waiting for them to enable support for these scenarios. I know that's definitely true for the CameraPreview which uses the MediaPlayerElement, but not sure what's supported or not 100% with the underlying MediaFrameReader api the CameraHelper uses.

I know in the docs here and here they're only calling out the CaptureCameraUI from the Windows.Media.Capture namespace, so not sure if this is a missing gap in documentation or their radar/planning.

I'd recommend opening a discussion on the Windows App SDK repo to be sure of what is and isn't supposed to be working at this time.

Thank you very much for your answer. About the media playback control and video preview control is not porting too difficult to become a low priority?

[michael-hawker]

Thank you very much for your answer. About the media playback control and video preview control is not porting too difficult to become a low priority?

Hi @GreenShadeZhang, I'm not 100% sure I'm following what you're asking here. The Windows App SDK team handles the underlying APIs we're using, so filing an issue on their repo and asking for clarity on what should be supported is a first step. They may clarify if it's supposed to work now, not supposed to work, or something they're planning to implement in an upcoming release or not.

From there, once we know that, we can figure out on our side in the Toolkit for our next 8.0 release if this is something we can support, assuming it is supported in the platform at that time.

[AlexanderBlackman]

I've had a similar problem, I guess it can wait until the MediaPlayerElement is updated in the SDK.

[GreenShadeZhang]

I've had a similar problem, I guess it can wait until the MediaPlayerElement is updated in the SDK.

Recently I replaced the camera API of UWP with opencvsharp, and the effect is OK is that the CPU occupation is too high, I don't know if you have any good way.

[GreenShadeZhang]

Thank you very much for your answer. About the media playback control and video preview control is not porting too difficult to become a low priority?

Hi @GreenShadeZhang, I'm not 100% sure I'm following what you're asking here. The Windows App SDK team handles the underlying APIs we're using, so filing an issue on their repo and asking for clarity on what should be supported is a first step. They may clarify if it's supposed to work now, not supposed to work, or something they're planning to implement in an upcoming release or not.

From there, once we know that, we can figure out on our side in the Toolkit for our next 8.0 release if this is something we can support, assuming it is supported in the platform at that time.

Recently I found out that the Windows app SDK 1.2 preview version supports MediaPlayerElement, so I feel that 8.0 can test the integrity of this feature. https://learn.microsoft.com/en-us/windows/apps/windows-app-sdk/preview-channel#winui-3

In addition, I also saw some precautions for camera data processing in the document, and I don't know if I need to pay attention to these things in the windows app SDK. https://learn.microsoft.com/en-us/windows/uwp/audio-video-camera/process-media-frames-with-mediaframereader

[michael-hawker]

Thanks @GreenShadeZhang, our big hope with 8.0 is to maintain everything from a single codebase (vs. the forks/branches we have now). So, ideally we'll be able to make this easier for us to validate the behavior of and test across both UWP and the Windows App SDK.

[GreenShadeZhang]

Thanks @GreenShadeZhang, our big hope with 8.0 is to maintain everything from a single codebase (vs. the forks/branches we have now). So, ideally we'll be able to make this easier for us to validate the behavior of and test across both UWP and the Windows App SDK.

Sounds very good and very exciting.