[Question] CPU usage skyrockets when using Media Foundation with Hardware Decoding

[charlesingea]

AVPro Video Version

AVPro Video v3 - Ultra Edition

Which platform is your question for?

Windows

Your question

Hello, I am building a project that requires several video tracks to be running concurrently. This is for a Stereo Active sequential display, which means that I am using Graphics.Blit to copy out the textures for each stereo eye onto the compositor used by my display. The videos I am playing are H265, top-bottom stereo. My MediaPlayers in AVPro are set up for Media Foundation and use Hardware Encoding.

When I play my project in the Unity Editor, it runs super smoothly and I can see in my Task Manager that my RTX GPU is being used for video decoding. But when I build my project and try to run it, the videos are very jerky and I can see in my Task Manager that my CPU is at 100% capacity while both my GPUs are not really working.

I don't know if this means that I am not using Hardware Decoding correctly? I am not sure how to debug this issue further. Any help is appreciated.

[Chris-RH]

1. Are you using built-in renderer, URP or HDRP?
2. D3D11 or D3D12?
3. Does the same occur using DirectShow and WinRT API?
4. Have you checked that the Quality settings are the same for in editor and build?
5. What are the specs of your videos? How many are you playing at once?

[charlesingea]

Hi Chris, thanks for your reply. See my answers below:

1. I am using the built-in renderer
2. Direct3D12. Changing to D3D11 didn't seem to make a difference.
3. With DirectShow, the videos appear to play a bit smoother (but still laggy), but the CPU usage is still 100%. With WinRT, the screen appears white, but I can see that my GPU is working on 'Video Decode' and the CPU usage appears normal.
4. Yes, the quality settings are the same.
5. I am playing 3 videos at once. They are H265, 4300x3200px, top-down stereo. Struggling to understand why they play back so well in the editor but not in a build! Thanks for your help.

[Chris-RH]

Is the resolution setting in the editor the same as in build? Have you checked your GPU's control panel? I wonder if its somehow auto selecting integrated GPU rather than discrete GPU?

[Chris-RH]

Is the resolution setting in the editor the same as in build? Have you checked your GPU's control panel? I wonder if its somehow auto selecting integrated GPU rather than discrete GPU?

[charlesingea]

Hi Chris, Thanks for getting back to me.

Yes, the resolution settings are the same in both editor and build. My settings are all set to default in the NVIDIA control p[anel - it doesn't look like it's selecting the integrated GPU because of that.

Any other idea of where the problem might come from would be greatly appreciated!

[Chris-RH]

What GPU are you using? Is this all connected to the projector mentioned in previous posts? Thoughts are that for some reason, Windows thinks that your GPU isn't connected to a display and is therefore ignoring it. You could try connecting something like this to the HDMI to see if it "tricks" Windows.

[charlesingea]

I have 2 GPUs connected: a RTX A6000 and a Quadro P400. Yes, I am using a stereo projector setup. I am using an app and Unity plugin called MiddleVR to act as the middleman between my Unity app and the projector setup. When I play the app in editor, I can see in the task manager that my RTX card is busy with "video decode" tasks. But when I play the built executable then this doesn't happen anymore and instead the CPU usage goes up to 100%.

I am struggling to understand why Unity would behave differently in this regard between the editor and a build?

Another thing worth noting is that I am also using a smaller monitor for UI purposes in addition to the stereo projector setup. But even if I deactivate it, I still get the same issue, so it most probably is unrelated.

[Chris-RH]

which Unity version are you using?

[charlesingea]

I am using 2023.2.20f1

[Chris-RH]

If you opened a new project, using just AVPro Video and the mediaplayer demo scene, do you get the same results using that hardware?

[charlesingea]

Hi Chris, I tried a new, clean project in Unity 6.1. The MediaPlayer Demo scene runs fine, and even my custom scene works fine when I use the demo media assets. So it seems to be a problem with my media specifically. To be fair, it is quite heavy media: I am trying to play a 4,300px x 3,200px top-down stereo H265 video. What I don't understand, though, is how the Unity Editor is able to breeze through 3 of these massive videos running simultaneously without even dropping a single frame (making the most of my RTX A6000), while a built .exe will consistently fail and max out my CPU, seemingly without utilising my GPUs like it should. Any idea on how to debug this is greatly appreciated!

[charlesingea]

Hi Chris,

Here is an extract from the log when I play my app in the editor:

[AVProVideo] Initialising AVPro Video v3.2.6 (native plugin v3.2.6f1-ultra) on NVIDIA RTX A6000/Direct3D 12 [level 12.1] (MT False) on WindowsEditor
[AVProVideo] Opening [file path] (offset 0) with API MediaFoundation
[AVProVideo] Using playback path: MF-MediaEngine-Hardware ([email protected])

And here is the same log when I play a build:

[AVProVideo] Initialising AVPro Video v3.2.6 (native plugin v3.2.6f1-ultra) on NVIDIA RTX A6000/Direct3D 12 [level 12.1] (MT False) on WindowsPlayer
[AVProVideo] Opening [file path] (offset 0) with API MediaFoundation
RenderGraph is now enabled.
[AVProVideo] Using playback path: MF-MediaEngine-Software ([email protected])
Registering a native texture with width=4320 while the actual texture has width=4300

AVPro seems to revert to software decoding in the build but uses hardware decoding in the editor. Any idea why this might happen? Also not totally sure why the resolution seems to mismatch? (4,300 versus 4,320)

[charlesingea]

I would just like to add that this also happens if I try to play one of the H264 AVPro media assets:

[AVProVideo] Initialising AVPro Video v3.2.6 (native plugin v3.2.6f1-ultra) on NVIDIA RTX A6000/Direct3D 12 [level 12.1] (MT False) on WindowsPlayer
[AVProVideo] Opening [project path]/StreamingAssets\AVProVideoSamples/RenderHeads-1080p30-H264.mp4 (offset 0) with API MediaFoundation
RenderGraph is now enabled.
[AVProVideo] Using playback path: MF-MediaEngine-Software ([email protected])

So it seems like AVPro consistently falls back to software decoding regardless of video codec?

[Chris-RH]

This is very strange. Are you able to provide a sample of the video(s) used for testing please? [email protected]

[charlesingea]

Sure, I'll email a sample over - but the same thing happens with AVPro's sample media (as per my last comment above!) I wonder if it is something to do with my scripting or the way I instantiate the MediaPlayer which causes it to use Hardware encoding in the editor but Software encoding in a build?

[Chris-RH]

Perhaps you could make a simple project that reproduces the issue, especially if you suspect that its your scripting, and send that too?

[charlesingea]

Sure, that's a good idea - I will send shortly

[charlesingea]

Hi Chris, I've sent a sample project to your email. I don't think my issue has to do with the media I am playing (as I also get the same issue with the demo media assets) and I also don't think it has to do with my scripts as I also experience it in a completely stripped down scene with nothing but a MediaPlayer component. It might just be something strange happening with my hardware but would be keen to hear what you think.

[Chris-RH]

Thanks for sending the project. As expected, it works fine on our hardware. It must be something to do with your hardware setup that making it fail to create a D3D11 device for hardware playback, and therefore fall back to software decoding. It may be worth looking into player command-line flags (-force-device-index), so that you can force it to use a specific GPU: https://docs.unity3d.com/6000.1/Documentation/Manual/PlayerCommandLineArguments.html

[charlesingea]

Thanks so much for your help Chris, will look into it!

[charlesingea]

Hi Chris,

Sorry to reopen this, but I've managed to narrow this problem down and what I found is a really strange issue. Hoping you can help me on this.

I use a Unity plugin called MiddleVR, which acts as a middleman between Unity and my stereoscopic projector setup. I am trying to use AVPro to build an interactive stereo video player for my projector setup. The issue I keep running into is that, in the Editor, AVPro uses Hardware Decoding normally, but in a build, it switches back to Software Decoding.

I tried a new, clean, fresh project, with a scene that only contains an AVPro Media Player: it worked fine and used Hardware Decoding in both the Editor and a build - great. Then I added the MiddleVR plugin to the Unity project without even changing anything to the scene or to the project settings (literally just added a new package from the Package Manager) - and suddenly the issue comes up, where Hardware Decoding is used normally in the Editor, but Software Decoding is used in a build.

I've been in touch with the developers at MiddleVR and I've sent them the two projects - they ran into the same issue. They combed over every folder and tried a few different things, but couldn't get to the bottom of this issue. They suggested reaching out to you again.

I'd be super grateful if you could have a look again and let me know if you have any idea what might be causing this. It is definitely a strange issue!

Best wishes Charles

[charlesingea]

Sorry, I realise I forgot to attach the projects I am talking about. Please see link below and let me know if you have any trouble accessing.

[Link removed - please do not post links to project, instead post them to [email protected]]

[Ste-RH]

Link removed - please do not post links to project, instead post them to [email protected]

[charlesingea]

Sure, will do.

[Ste-RH]

I grabbed them already, so you don't have to post up on this occasion @charlesingea :)

[Reuben-RH]

Hi @charlesingea

The issue appears to be caused by a build post process step done by MiddleVR, causing the AVPro .manifest file to be removed from the final .exe (this is required for version checks done by AVPro Video).

I have attached a .zip file containing the Settings.manifest that AVProVideo .dlls are made with, along with a Windows Batch file that will combine the current manifest of the .exe with this manifest. This is done using Microsoft (R) Manifest tool.

ManifestMerge.zip

To Use:

• Place the "Settings.manifest" && "CombineManifest.bat" files within the same folder as the .exe
  • ensure there are no other .exe files within the folder (bat file just inserts manifest into first .exe file)
• Run the CombineManifest.bat file (double click)
• and it should combine the manifests into the .exe

The state of the manifest can be found by running the command mt.exe -inputresource:{EXECUTABLE_NAME}.exe;#1 -out:manifest.xml

Alternatively the manifest of the .exe can be fully replaced by performing the following command: mt.exe -manifest Settings.manifest -outputresource:"{EXECUTABLE_NAME}.exe;#1"

for more information about mt.exe please see: https://learn.microsoft.com/en-us/windows/win32/sbscs/mt-exe

Please let us know if this fixes the issue for you.

RBN