[Performance]: RAR is 1.8x slower in MSBuild.exe compared to dotnet build

[ladipro]

There's a significant difference between Framework and Core MSBuild in the time taken up by the ResolveAssemblyReference when building OrchardCore.

Details:

• Upstream OrchardCore at commit 2804ec773.
• MSBuild 17.9 candidate (commit 5fcddc790 but unlikely to make a difference)
• Command lines (after doing a restore and build):
  • dotnet build --no-restore /bl
  • msbuild /m /bl

Expected output

• Understand what makes MSBuild.exe slower. Are some of the optimizations introduced in 8.0 not kicking in?
• Fix (or suggest a fix; depending on complexity) the perf gap.

[YuliiaKovalova]

After conducting an investigation, a couple of weak spots were identified:

• A slow function call within mscorlib.SystemIO - Microsoft.IO.Redist was utilized to address this performance gap https://github.com/dotnet/msbuild/pull/9814.
• TaskItem.CopyMetadataTo is slower in msbuild.exe due to the underlying system.collections.immutable - can we do anything here @rainersigwald ? Update: it was discussed offline - one of the potential strategy is to rethink the way we set metadata here https://github.com/dotnet/msbuild/blob/main/src/Tasks/AssemblyDependency/ReferenceTable.cs#L2674

• Invoking SystemState.FileExists in an executable is more costly due to underlying PInvoke operations.

update Running on amd64 gives a noticeable boost!

• LogResults invocation is very different in msbuild.exe, it goes through CentralForwardingLogger.EventSource_AnyEventRaised The idea that came to Rainer's mind is related to making logging async, because now it looks like a bottleneck in RAR. The reason why it is implemented this way for a single node invocation: https://github.com/dotnet/msbuild/blob/67f702b535a02be05a696376883bd9cd6d576e38/src/Build/BackEnd/BuildManager/BuildParameters.cs#L320-L324 https://github.com/dotnet/msbuild/blob/67f702b535a02be05a696376883bd9cd6d576e38/src/MSBuild/XMake.cs#L1428-L1433 After enabling MSBUILDLOGASYNC=1 we received an immediate perf boost! We have to decide on switching it by default.

The snapshot used for comparison:

[YuliiaKovalova]

The investigation is done and conclusions are described above. The potential work items is created for async logging + optimization for File traversal was applied.