microsoft
Test discovery broken on Android from 2.2.10+
Describe the bug
We have a MAUI test runner that we deploy to both Android and iOS phones. It is using (MSTestX.UnitTestRunner 2.0.0-preview4). The project is configured with: <TargetFrameworks>net6.0-android;net6.0-ios</TargetFrameworks> using 'android'">21.0.
When we include MSTest.TestFramework 2.2.9 in the referenced test projects that contains the tests, all tests are discovered correctly on BOTH iOS and Android
When we update MSTest.TestFramework to 2.2.10 or 3.x.y in the referenced test projects that contains the tests, all tests are discovered correctly ONLY on iOS, but fails on android.
Looking at the code in repo, I can see that there are significant changes in test discovery code from 2.2.9 -> 2.2.10 in MSTest.TestFramework. I hope you will try testing this on android.
Steps To Reproduce
- Create a test project targeting <TargetFrameworks>net6.0-android;net6.0-ios</TargetFrameworks>
- Add some MS tests that are data driven. We use the following test attributes: DataTestMethodAttribute, DataRowAttribute
- reference MSTest.TestFramework 2.2.9
- create a maui testrunner application targeting <TargetFrameworks>net6.0-android;net6.0-ios</TargetFrameworks>
- reference the above test project
- reference MSTestX.UnitTestRunner 2.0.0-preview4
-
Run on android phone and see discovery of tests work
-
update the test proejct to reference MSTest.TestFramework 2.2.10
-
Run on android phone and see discovery of tests no longer works (0 tests discovered)
Expected behavior
Test discovery works on android for .net60 test projects.
Actual behavior
0 tests are discovered on android.
Hello @MichaelRostgaard, Can you give us a simple project with the issue reproducible?
Hi,
I will provide you with a sample project as soon as possible (I may be able to prioritize this within a few weeks).
/Michael
From: Enji Eid @.> Sent: 14. marts 2023 11:35 To: microsoft/testfx @.> Cc: Michael Rostgaard (MHRT) @.>; Mention @.> Subject: Re: [microsoft/testfx] Test discovery broken on Android from 2.2.10+ (Issue #1617)
This message comes from outside of the corporate network. Please do not click links or open attachments unless you recognize the source of this e-mail and know the content is safe.
Hello @MichaelRostgaardhttps://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMichaelRostgaard&data=05%7C01%7Cmhrt%40demant.com%7C42528344d0b246ea880e08db2477d57c%7C9bf8c7a8e00849a79e43ab76976c4bf8%7C0%7C0%7C638143869320590801%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=QKbr%2FzOp2gECgB2HJ%2FgBfDq%2BWYwJ8%2Fs%2FGxpejiR%2BFwo%3D&reserved=0, Can you give us a simple project with the issue reproducible?
Reply to this email directly, view it on GitHubhttps://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmicrosoft%2Ftestfx%2Fissues%2F1617%23issuecomment-1467836954&data=05%7C01%7Cmhrt%40demant.com%7C42528344d0b246ea880e08db2477d57c%7C9bf8c7a8e00849a79e43ab76976c4bf8%7C0%7C0%7C638143869320590801%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=WbnB9xy3mOuseKUqOHGao6NSBTSIeU028pD8Se25KuE%3D&reserved=0, or unsubscribehttps://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FA6HKTOL5IRPNLBRHQ4YXBODW4BCXDANCNFSM6AAAAAAVQ2CBVA&data=05%7C01%7Cmhrt%40demant.com%7C42528344d0b246ea880e08db2477d57c%7C9bf8c7a8e00849a79e43ab76976c4bf8%7C0%7C0%7C638143869320590801%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=04FBcIvzgFh6SEbyLEyi2Pg%2BFQIl0MSzvm4vrN1ncS0%3D&reserved=0. You are receiving this because you were mentioned.Message ID: @.@.>>
Sample embedded.
- Unpack the rar file-
- Open MSTestFrameworkErrorOnAndroid.sln in Visual studio (I use Visualstudio professional 2022 version 17.4.4)
- Set IntegrationTest.TestRunner.Net6 as startup project, and connect an android phone with debug permission enabled
- Compile and deploy to android phone. -> Verify 3 tests detected on phone (2 passed, 1 failed), stop app
- Edit IntegrationTest.Net6.Mobile.csproj -> change:
To
- Compile and deploy to android phone. -> Verify 0 tests detected on phone
/Michael
From: Enji Eid @.> Sent: 14. marts 2023 11:35 To: microsoft/testfx @.> Cc: Michael Rostgaard (MHRT) @.>; Mention @.> Subject: Re: [microsoft/testfx] Test discovery broken on Android from 2.2.10+ (Issue #1617)
This message comes from outside of the corporate network. Please do not click links or open attachments unless you recognize the source of this e-mail and know the content is safe.
Hello @MichaelRostgaardhttps://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMichaelRostgaard&data=05%7C01%7Cmhrt%40demant.com%7C42528344d0b246ea880e08db2477d57c%7C9bf8c7a8e00849a79e43ab76976c4bf8%7C0%7C0%7C638143869320590801%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=QKbr%2FzOp2gECgB2HJ%2FgBfDq%2BWYwJ8%2Fs%2FGxpejiR%2BFwo%3D&reserved=0, Can you give us a simple project with the issue reproducible?
Reply to this email directly, view it on GitHubhttps://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmicrosoft%2Ftestfx%2Fissues%2F1617%23issuecomment-1467836954&data=05%7C01%7Cmhrt%40demant.com%7C42528344d0b246ea880e08db2477d57c%7C9bf8c7a8e00849a79e43ab76976c4bf8%7C0%7C0%7C638143869320590801%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=WbnB9xy3mOuseKUqOHGao6NSBTSIeU028pD8Se25KuE%3D&reserved=0, or unsubscribehttps://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FA6HKTOL5IRPNLBRHQ4YXBODW4BCXDANCNFSM6AAAAAAVQ2CBVA&data=05%7C01%7Cmhrt%40demant.com%7C42528344d0b246ea880e08db2477d57c%7C9bf8c7a8e00849a79e43ab76976c4bf8%7C0%7C0%7C638143869320590801%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=04FBcIvzgFh6SEbyLEyi2Pg%2BFQIl0MSzvm4vrN1ncS0%3D&reserved=0. You are receiving this because you were mentioned.Message ID: @.@.>>
Hi @MichaelRostgaard,
I think you forget to attach the rar file.
and does the issue happen with you in latest version mstest3.0.2?
Sample was included... maybe the proof.rar file was removed by email service?
See image of my outbox @.***
From: Enji Eid @.@.> Sent: 17. marts 2023 13:02 To: microsoft/testfx @.> Cc: Michael Rostgaard (MHRT) @.>; Mention @.***> Subject: Re: [microsoft/testfx] Test discovery broken on Android from 2.2.10+ (Issue #1617)
This message comes from outside of the corporate network. Please do not click links or open attachments unless you recognize the source of this e-mail and know the content is safe.
I think you forget to attach the rar file.
and does the issue happen with you in latest version mstest3.0.2?
Reply to this email directly, view it on GitHubhttps://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmicrosoft%2Ftestfx%2Fissues%2F1617%23issuecomment-1473728292&data=05%7C01%7Cmhrt%40demant.com%7C613edde0ecb24e8d4e8708db26df7522%7C9bf8c7a8e00849a79e43ab76976c4bf8%7C0%7C0%7C638146513405484209%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=da65GI5jNli%2B%2FgSIHd1wqODQ9k8d02Izy7s2Ytf8jK8%3D&reserved=0, or unsubscribehttps://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FA6HKTONYNWAOWMY4TD3GNVTW4RHETANCNFSM6AAAAAAVQ2CBVA&data=05%7C01%7Cmhrt%40demant.com%7C613edde0ecb24e8d4e8708db26df7522%7C9bf8c7a8e00849a79e43ab76976c4bf8%7C0%7C0%7C638146513405484209%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=UxFe58LGHtyzkhPHH6Emr0HqYpJQf34T%2FAXJpGRldAQ%3D&reserved=0. You are receiving this because you were mentioned.Message ID: @.@.>>
and does the issue happen with you in latest version mstest3.0.2? Yes
From: Enji Eid @.> Sent: 17. marts 2023 13:02 To: microsoft/testfx @.> Cc: Michael Rostgaard (MHRT) @.>; Mention @.> Subject: Re: [microsoft/testfx] Test discovery broken on Android from 2.2.10+ (Issue #1617)
This message comes from outside of the corporate network. Please do not click links or open attachments unless you recognize the source of this e-mail and know the content is safe.
I think you forget to attach the rar file.
and does the issue happen with you in latest version mstest3.0.2?
Reply to this email directly, view it on GitHubhttps://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmicrosoft%2Ftestfx%2Fissues%2F1617%23issuecomment-1473728292&data=05%7C01%7Cmhrt%40demant.com%7C613edde0ecb24e8d4e8708db26df7522%7C9bf8c7a8e00849a79e43ab76976c4bf8%7C0%7C0%7C638146513405484209%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=da65GI5jNli%2B%2FgSIHd1wqODQ9k8d02Izy7s2Ytf8jK8%3D&reserved=0, or unsubscribehttps://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FA6HKTONYNWAOWMY4TD3GNVTW4RHETANCNFSM6AAAAAAVQ2CBVA&data=05%7C01%7Cmhrt%40demant.com%7C613edde0ecb24e8d4e8708db26df7522%7C9bf8c7a8e00849a79e43ab76976c4bf8%7C0%7C0%7C638146513405484209%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=UxFe58LGHtyzkhPHH6Emr0HqYpJQf34T%2FAXJpGRldAQ%3D&reserved=0. You are receiving this because you were mentioned.Message ID: @.@.>>
@MichaelRostgaard may be can you add it as a comment in the issue on github please
Hi @MichaelRostgaard,
I cannot compile the sample project you have provided. Looking at my installed workloads/components, I cannot see something missing, could you double check this is a runnable sample?
Also, looking at the runner, it seems to be something provided by someone else so it's possible this runner is not compatible with newer version of MSTest. This is not something we are testing or supporting so I would suggest to contact the maintainer.
I downloaded the sample I provided, unpacked it to a new folder. deleted my nuget packages folder (C:\Users[myusername].nuget\packages) to make sure all packages would be restored correct unpon compilation. Opened the solution in Visual studio 17.4.4. Compiled it and ran the project successfully on my phone.
So - yearh - I think the solution is valid.
/Michael