[Bug]: CustomAnalyzerTest tests fail on macOS 13 and 14

[jrdodds]

Issue Description

Recently PR #10076 added a new CustomAnalyzerTest [Theory] with two sets of data. The PR build checks passed running on a macos-12 image.

However, when I build locally and run tests on macOS 14 on ARM and macOS 13 on Intel, the tests fail.

Steps to Reproduce

1. Run ./build.sh.
2. Run ./build.sh --test

Expected Behavior

All tests should pass.

Actual Behavior

CustomAnalyzerTest tests fail.

Failed tests
Microsoft.Build.BuildCheck.UnitTests.EndToEndTests.CustomAnalyzerTest(customAnalyzerNames: [\"CustomAnalyzer\", \"CustomAnalyzer2\"], analysisCandidate: \"AnalysisCandidateWithMultipleAnalyzersInjected\", expectedRegisteredRules: [\"CustomRule1\", \"CustomRule2\", \"CustomRule3\"])
Shouldly.ShouldAssertException : success\r\n    should be\r\nTrue\r\n    but was\r\nFalse
   at Microsoft.Build.BuildCheck.UnitTests.EndToEndTests.BuildAnalyzerRules(TestEnvironment env, String[] customAnalyzerNames) in /Users/jrdodds/Projects/dotnet/msbuild/src/BuildCheck.UnitTests/EndToEndTests.cs:line 174
   at Microsoft.Build.BuildCheck.UnitTests.EndToEndTests.CustomAnalyzerTest(String[] customAnalyzerNames, String analysisCandidate, String[] expectedRegisteredRules) in /Users/jrdodds/Projects/dotnet/msbuild/src/BuildCheck.UnitTests/EndToEndTests.cs:line 145
   at InvokeStub_EndToEndTests.CustomAnalyzerTest(Object, Span`1)
   at System.Reflection.MethodBaseInvoker.InvokeWithFewArgs(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
Microsoft.Build.BuildCheck.UnitTests.EndToEndTests.CustomAnalyzerTest(customAnalyzerNames: [\"CustomAnalyzer\"], analysisCandidate: \"AnalysisCandidate\", expectedRegisteredRules: [\"CustomRule1\", \"CustomRule2\"])
Shouldly.ShouldAssertException : success\r\n    should be\r\nTrue\r\n    but was\r\nFalse
   at Microsoft.Build.BuildCheck.UnitTests.EndToEndTests.BuildAnalyzerRules(TestEnvironment env, String[] customAnalyzerNames) in /Users/jrdodds/Projects/dotnet/msbuild/src/BuildCheck.UnitTests/EndToEndTests.cs:line 174
   at Microsoft.Build.BuildCheck.UnitTests.EndToEndTests.CustomAnalyzerTest(String[] customAnalyzerNames, String analysisCandidate, String[] expectedRegisteredRules) in /Users/jrdodds/Projects/dotnet/msbuild/src/BuildCheck.UnitTests/EndToEndTests.cs:line 145
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodBaseInvoker.InvokeDirectByRefWithFewArgs(Object obj, Span`1 copyOfArgs, BindingFlags invokeAttr)

The Microsoft.Build.BuildCheck.UnitTests_net8.0_x64.html and .Microsoft.Build.BuildCheck.UnitTests_net8.0_x64xml log files are attached in a .zip.

Microsoft.Build.BuildCheck.UnitTests_net8.0_x64.zip

Analysis

The tests fail in BuildAnalyzerRules() after calling RunnerUtilities.ExecBootstrapedMSBuild():

https://github.com/dotnet/msbuild/blob/393c2fea652873416c8a2028810932a4fa94403f/src/BuildCheck.UnitTests/EndToEndTests.cs#L164-L174

Versions & Configurations

MSBuild version 17.11.0-dev-24263-01+a8e224f80 for .NET 17.11.0.26301

macOS Sonoma 14.4.1 Apple M1 Pro (ARM)

macOS Ventura 13.6.6 Intel Core i7

[YuliiaKovalova]

Hi @jrdodds,

Thank you for reporting the bug! I have recently merged the changed version for these tests: #10117 Could you please check it out?

[jrdodds]

Hi @YuliiaKovalova, The tests still fail. The log files are attached.

Microsoft.Build.BuildCheck.UnitTests_net8.0_x64.zip

MSBuild version 17.11.0-dev-24304-01+db79545e5 for .NET 17.11.0.3040

macOS Sonoma 14.5 Apple M1 Pro (ARM)

[YuliiaKovalova]

@jrdodds, It has to be resolved now, after merging https://github.com/dotnet/msbuild/issues/10300

Please validate it when you have a chance .

[jrdodds]

@YuliiaKovalova Tested/validated and confirmed that the issue is resolved. Thanks

[YuliiaKovalova]

@jrdodds , thank you for checking it! Your issue got it inspired to revisit the approach to bootstrapping MSBuild and testing the instance. Feel free to dive deeper if you are interested: https://github.com/dotnet/msbuild/blob/main/documentation/wiki/Bootstrap.md