microsoft
PosterCalibration Camera Not Working in Unity 2018
I have been working on updating the PosterCalibration Sample to Unity 2018. I have everything compiling, I have all gestures working with the new APIs and I have voice commands working. However, I cannot get the camera feed to work and therefore posters are not detected.
My fork (and branch) for this work can be found here:
https://github.com/jbienzms/MixedRealityCompanionKit/tree/PosterUnity2018
The code deploys to the HoloLens and mostly runs, but when you say "Align Poster 1" the camera feed is just a white square. When connected to the Visual Studio debugger, the following is printed in the Output window:
AlignZone: CalibrationZone
(Filename: C:\buildslave\unity\build\Runtime/Export/Debug.bindings.h Line: 43)
ImagePosterLocator:SetPosterTexture Mips:1
(Filename: C:\buildslave\unity\build\Runtime/Export/Debug.bindings.h Line: 43)
ImagePosterLocator:SetPosterTexture - using mip #0:1024x622
(Filename: C:\buildslave\unity\build\Runtime/Export/Debug.bindings.h Line: 43)
ImagePosterLocator:StartProcessing: CalibrationZone
(Filename: C:\buildslave\unity\build\Runtime/Export/Debug.bindings.h Line: 43)
Update version updated.
(Filename: C:\buildslave\unity\build\Runtime/Export/Debug.bindings.h Line: 43)
PhotoCapture created...
(Filename: C:\buildslave\unity\build\Runtime/Export/Debug.bindings.h Line: 43)
Photo Mode has already been started.
(Filename: C:\buildslave\unity\build\Runtime/VR/HoloLens/WebCam/PhotoCapture.cpp Line: 530)
Exception thrown: 'System.NotImplementedException' in WinRTBridge.dll
'PosterCalibrationSample.exe' (CoreCLR: CoreCLR_UWP_Domain): Loaded 'C:\Data\Users\DefaultAccount\AppData\Local\DevelopmentFiles\PosterCalibrationSampleVS.Release_x86.jbienz\System.Diagnostics.StackTrace.dll'. Cannot find or open the PDB file.
Exception: The method or operation is not implemented.
Type: System.NotImplementedException
Module: WinRTBridge
InnerException: <No Data>
AdditionalInfo:MarshalFirstFieldIntoNativePtr type: UnityEngine.XR.WSA.WebCam.PhotoCaptureFrame
at WinRTBridge.Marshalling.MarshalFirstFieldIntoNativePtr(Object instance)
(Filename: Line: 0)
Exception thrown: 'System.ArgumentNullException' in UnityEngineProxy.dll
ArgumentNullException: Value cannot be null.
Parameter name: _unity_self
at UnityEngineProxy.InternalCalls.PhotoCaptureFrame_CUSTOM_GetDataLength(Object self)
at UnityEngine.XR.WSA.WebCam.PhotoCaptureFrame..ctor(IntPtr nativePtr)
at UnityEngine.XR.WSA.WebCam.PhotoCapture.InvokeOnCapturedPhotoToMemoryDelegate(OnCapturedToMemoryCallback callback, Int64 hResult, IntPtr photoCaptureFramePtr)
at UnityEngine.XR.WSA.WebCam.PhotoCapture.$Invoke10(Int64 instance, Int64* args)
at UnityEngine.Internal.$MethodUtility.InvokeMethod(Int64 instance, Int64* args, IntPtr method)
(Filename: <Unknown> Line: 0)
@dberrett Hey Doug. Any thoughts on this? Is this happening in the unmanaged plugin? The plugin still compiles, but it appears to not be getting the expected pointer back, and a NotImplementedException seems rather unexpected.
The signature of the PhotoCaptureFrame has changed also. I think they removed the Boolean at the beginning of the parameters.
From: Jared Bienz [MSFT] [email protected] Sent: Tuesday, August 14, 2018 1:03 PM To: Microsoft/MixedRealityCompanionKit [email protected] Cc: Doug Berrett [email protected]; Mention [email protected] Subject: Re: [Microsoft/MixedRealityCompanionKit] PosterCalibration Camera Not Working in Unity 2018 (#299)
@dberretthttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fdberrett&data=02%7C01%7Cdberrett%40microsoft.com%7C68f138d84456467df06108d60220ebef%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636698737752155170&sdata=c9jnNERKFWsrqi6iAidBpGUJjMzeBreUvUT9TNLxpGw%3D&reserved=0 Hey Doug. Any thoughts on this? Is this happening in the unmanaged plugin? The plugin still compiles, but it appears to not be getting the expected pointer back, and a NotImplementedException seems rather unexpected.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMicrosoft%2FMixedRealityCompanionKit%2Fissues%2F299%23issuecomment-412996715&data=02%7C01%7Cdberrett%40microsoft.com%7C68f138d84456467df06108d60220ebef%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636698737752165174&sdata=djMGisfN9F8bgdqK1r09%2FOffcyeBJEwEaLxy5pr4hxA%3D&reserved=0, or mute the threadhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAHSBQA4gPJzyTEQKX34VGxoD1-WKNNoPks5uQyztgaJpZM4V9FCV&data=02%7C01%7Cdberrett%40microsoft.com%7C68f138d84456467df06108d60220ebef%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636698737752165174&sdata=cC6GTer36HlEK5shb%2B5tuNSRAokgioB5%2B6yTCFg8EkU%3D&reserved=0.
Forgive my ignorance on this one, but why is the C++ plugin project compiling correctly if the method signature changed?
That’s Unity that is failing, correct? Not the C++ plugin. Unity takes the photo and passes the image data to the plugin. In Unity 2017.x the method signature of the photo capture changed. I don’t recall the exact change, but I think it was just removing a Boolean on the front of one of the calls. Adding @Troy Ferrellmailto:[email protected] who recently ran this successfully on Unity 2018.
From: Jared Bienz [MSFT] [email protected] Sent: Wednesday, August 15, 2018 7:06 AM To: Microsoft/MixedRealityCompanionKit [email protected] Cc: Doug Berrett [email protected]; Mention [email protected] Subject: Re: [Microsoft/MixedRealityCompanionKit] PosterCalibration Camera Not Working in Unity 2018 (#299)
Forgive my ignorance on this one, but why is the C++ plugin project compiling correctly if the method signature changed?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMicrosoft%2FMixedRealityCompanionKit%2Fissues%2F299%23issuecomment-413207972&data=02%7C01%7Cdberrett%40microsoft.com%7C88c55e7e06214c7ed51a08d602b83519%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636699387527823765&sdata=gUCELieuHSndy97KWzkTTV7VwcMpjaIX6c9Y7ojX8nQ%3D&reserved=0, or mute the threadhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAHSBQLqjHNNhy8am6cGaRRKtuKga3dg5ks5uRCq4gaJpZM4V9FCV&data=02%7C01%7Cdberrett%40microsoft.com%7C88c55e7e06214c7ed51a08d602b83519%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636699387527823765&sdata=mi3JBtnGErwarpkP2vsO2SE8nw9y6XyZKFDwnM35754%3D&reserved=0.
Actually, I was pretty sure this was in the C++ code. I could be wrong, but the first exception is the System.NotImplementedException which at least appears to have occurred in PosterCalibrationSampleVS.Release_x86.
AdditionalInfo:MarshalFirstFieldIntoNativePtr type: UnityEngine.XR.WSA.WebCam.PhotoCaptureFrame
I could be wrong about that, but that's the info we're getting in the output window.
Maybe I’m missing something. The “NotImplementedException” was in “PosterCalibrationSampleVS.Release_x86” – which is the Unity VS project. And this line seems to indicate a failure in the Unity engine when it goes to make a capture and marshal the parameters to the WinRTBridge - not the C++ plugin side.
AdditionalInfo:MarshalFirstFieldIntoNativePtr type: UnityEngine.XR.WSA.WebCam.PhotoCaptureFrame
From: Jared Bienz [MSFT] [email protected] Sent: Wednesday, August 15, 2018 8:38 AM To: Microsoft/MixedRealityCompanionKit [email protected] Cc: Doug Berrett [email protected]; Mention [email protected] Subject: Re: [Microsoft/MixedRealityCompanionKit] PosterCalibration Camera Not Working in Unity 2018 (#299)
Actually, I was pretty sure this was in the C++ code. I could be wrong, but the first exception is the System.NotImplementedException which at least appears to have occurred in PosterCalibrationSampleVS.Release_x86.
AdditionalInfo:MarshalFirstFieldIntoNativePtr type: UnityEngine.XR.WSA.WebCam.PhotoCaptureFrame
I could be wrong about that, but that's the info we're getting in the output window.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMicrosoft%2FMixedRealityCompanionKit%2Fissues%2F299%23issuecomment-413237407&data=02%7C01%7Cdberrett%40microsoft.com%7C39c183dac15f497ff5e108d602c51df1%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636699442968175111&sdata=XwBlY1izngdihARpKVfaJmRMaXqrCCykqeE29g5JJrw%3D&reserved=0, or mute the threadhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAHSBQO8geofVoADxrhzM9SWlsU70tSddks5uREBmgaJpZM4V9FCV&data=02%7C01%7Cdberrett%40microsoft.com%7C39c183dac15f497ff5e108d602c51df1%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636699442968175111&sdata=0x7wtFqqH3n%2Boz7prX6COAUGUgpAO5q1s%2FcY0HhN2Mc%3D&reserved=0.
No, you're right @dberrett . For some reason I was thinking the VS project was the C++ project but you're right that it's the Unity project exported to VS. Is this a new Unity bug that we need to open with them?
Before we decide that for sure, I just want to make sure we're not doing something we shouldn't be. I noticed this message as well:
Photo Mode has already been started.
Perhaps there is a bug trying to initialize the camera twice?
We just tested this a couple weeks ago and it worked, so I’m not sure what might have changed. I’ll try to take a look when I can, but not sure when I’ll get to it.
From: Jared Bienz [MSFT] [email protected] Sent: Wednesday, August 15, 2018 9:10 AM To: Microsoft/MixedRealityCompanionKit [email protected] Cc: Doug Berrett [email protected]; Mention [email protected] Subject: Re: [Microsoft/MixedRealityCompanionKit] PosterCalibration Camera Not Working in Unity 2018 (#299)
No, you're right @dberretthttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fdberrett&data=02%7C01%7Cdberrett%40microsoft.com%7Ccc63b7c5b51b4eabaae808d602c99dd5%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636699462299021992&sdata=37A%2FxU0eHdfnQoeHUhB7Klxc45zjqj5ZkZG9w%2F74rzE%3D&reserved=0 . For some reason I was thinking the VS project was the C++ project but you're right that it's the Unity project exported to VS. Is this a new Unity bug that we need to open with them?
Before we decide that for sure, I just want to make sure we're not doing something we shouldn't be. I noticed this message as well:
Photo Mode has already been started.
Perhaps there is a bug trying to initialize the camera twice?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMicrosoft%2FMixedRealityCompanionKit%2Fissues%2F299%23issuecomment-413247053&data=02%7C01%7Cdberrett%40microsoft.com%7Ccc63b7c5b51b4eabaae808d602c99dd5%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636699462299032001&sdata=%2FT4Dz259%2BKblgVvLE9xZFH%2FFrsaQSuCqUBhzYGEq8YM%3D&reserved=0, or mute the threadhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAHSBQG4lTaePWbPo6epoe2Sv0fvTdUe2ks5uREfygaJpZM4V9FCV&data=02%7C01%7Cdberrett%40microsoft.com%7Ccc63b7c5b51b4eabaae808d602c99dd5%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636699462299032001&sdata=8cxOugX%2BbvRJrn21hXeb9CSLE2JASTcUbmvHw5LX26Y%3D&reserved=0.
@jbienzms : I think this is a Unity problem. I have been trying to resolve this for few days. Last successful line of code is photoCaptureObject.TakePhotoAsync(OnCapturedPhotoToMemory). But it actually never enters the OnCapturedPhotoToMemory method. After figuring out where it is failing, I copied unity sample exactly as shown in their website https://docs.unity3d.com/Manual/windowsholographic-photocapture.html and it still fails. This happened when I tried to upgrade my project to Unity 2018.2. I tried to set it to IL2CPP and it compiled correctly, but it failed displaying results when I deployed it to the device. I assumed it was azure initially, but I was not able to debug it using C++. I tried to attach to debug installed app package to be able to debug in C#, I connected successfully, but failed to hit any break points and debugging this issue using C++ was painful for me. Finally, after so many days, I decided to switch it back to .NET backend and debugged the project to come to the conclusion that it is a unity problem. Tomorrow, I will try the beta version and keep trying prior versions to determine when is the last working version. Side note: Do you know how to debug the C# project when you use IL2CPP backend? I know how make it compile correctly, but not debug it. It fails to attach unity debugger to the hololens device. But visual studio allows me to attach debugger to the deployed application, but it will not hit the C# code, I chose managed and then i switched to native, then switched to both and none worked. According to this video from unity, it should work: https://oc.unity3d.com/index.php/s/rx7KD0SYeQXr6qn and this is the unity thread that I got the video from https://oc.unity3d.com/index.php/s/rx7KD0SYeQXr6qn
I will update you when I figure out something.
Hi all,
I was able to get the project working in Unity 2018.1.9f2 and HoloLens April 2018.
I did the following to get it working:
- Update the obsolete interaction APIs (i.e from "SourcePressed" to "InteractionSourcePressed").
- Ensure ImagePosterLocator component on CalibrationZone object is hooked up to PVCamView & PosterDetectView gameobject appropriately
- AutoAlignToPoster has Poster gameobject set correctly
- SpeechCommands component/gameobject is connected to the CalibrationZone (ZoneCalibrationManager)
- Only build the single-zone scene
I have the configuration for building as follows (under PlayerSettings/OtherSettings): Scripting Runtime Version: .Net 3.5 equivalent Scripting Backend: .NET API Compatibility Level: .Net 4.x
Further, make sure you have capabilities enabled for WebCam and Microphone under PlayerSettings/Publishing Settings
Try to match to this parity.
Thanks, Troy
@Troy-Ferrell Thank you for your help. My issue is a little different, I was trying to use the camera directly to take photos. I apologize for posting my issue here. I saw that @jbienzms has the same camera problem even though we are doing two different things.
- I created a workaround for the camera using MediaCapture: https://github.com/MSAlshair/HoloLensMediaCapture
- FYI: There is a Unity public active tracker: https://issuetracker.unity3d.com/issues/windowsmr-failure-to-take-photo-capture-in-hololens Thank you again for your help!
Thank you so much @MSAlshair for your workaround and taking the time to publish a sample. By any chance have you submitted this bug to Unity? And if not, any chance you could submit it? We'd like to know the bug number so we can start tracking it.
@jbienzms : You are very welcome. I didn't create a bug with Unity, but as I stated in my last post, I found an active bug that was already submitted since April 2018 and it is still open
Unity Bug:
https://issuetracker.unity3d.com/issues/windowsmr-failure-to-take-photo-capture-in-hololens
Unity forms discussing same issue:
https://forum.unity.com/threads/hololens-photo-capturing-failing.548845/
https://forum.unity.com/threads/photocapture-not-called-while-running-in-hololens.541825/