[Bug]: ObjectDisposedException in BuildManager.BuildGraph

[dfederm]

Issue Description

A customer is seeing an exception when building with /graph:

Unhandled exception. System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'Microsoft.Win32.SafeHandles.SafeWaitHandle'.
   at Interop.Kernel32.SetEvent(SafeWaitHandle handle)
   at System.Threading.EventWaitHandle.Set()
   at Microsoft.Build.Execution.BuildManager.<>c__DisplayClass99_0.<BuildGraph>b__2(BuildSubmission finishedBuildSubmission)
   at Microsoft.Build.Shared.ThreadPoolExtensions.<>c__DisplayClass0_0.<QueueThreadPoolWorkItemWithCulture>b__0(Object state)
   at System.Threading.QueueUserWorkItemCallback.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
Unhandled exception. System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'Microsoft.Win32.SafeHandles.SafeWaitHandle'.
   at Interop.Kernel32.SetEvent(SafeWaitHandle handle)
   at System.Threading.EventWaitHandle.Set()
   at Microsoft.Build.Execution.BuildManager.<>c__DisplayClass99_0.<BuildGraph>b__2(BuildSubmission finishedBuildSubmission)
   at Microsoft.Build.Shared.ThreadPoolExtensions.<>c__DisplayClass0_0.<QueueThreadPoolWorkItemWithCulture>b__0(Object state)
   at System.Threading.QueueUserWorkItemCallback.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
Unhandled exception. System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'Microsoft.Win32.SafeHandles.SafeWaitHandle'.
   at Interop.Kernel32.SetEvent(SafeWaitHandle handle)
   at System.Threading.EventWaitHandle.Set()
   at Microsoft.Build.Execution.BuildManager.<>c__DisplayClass99_0.<BuildGraph>b__2(BuildSubmission finishedBuildSubmission)
   at Microsoft.Build.Shared.ThreadPoolExtensions.<>c__DisplayClass0_0.<QueueThreadPoolWorkItemWithCulture>b__0(Object state)
   at System.Threading.QueueUserWorkItemCallback.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
Unhandled exception. System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'Microsoft.Win32.SafeHandles.SafeWaitHandle'.
   at Interop.Kernel32.SetEvent(SafeWaitHandle handle)
   at System.Threading.EventWaitHandle.Set()
   at Microsoft.Build.Execution.BuildManager.<>c__DisplayClass99_0.<BuildGraph>b__2(BuildSubmission finishedBuildSubmission)
   at Microsoft.Build.Shared.ThreadPoolExtensions.<>c__DisplayClass0_0.<QueueThreadPoolWorkItemWithCulture>b__0(Object state)
   at System.Threading.QueueUserWorkItemCallback.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
Unhandled exception. System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'Microsoft.Win32.SafeHandles.SafeWaitHandle'.
   at Interop.Kernel32.SetEvent(SafeWaitHandle handle)
   at System.Threading.EventWaitHandle.Set()
   at Microsoft.Build.Execution.BuildManager.<>c__DisplayClass99_0.<BuildGraph>b__2(BuildSubmission finishedBuildSubmission)
   at Microsoft.Build.Shared.ThreadPoolExtensions.<>c__DisplayClass0_0.<QueueThreadPoolWorkItemWithCulture>b__0(Object state)
   at System.Threading.QueueUserWorkItemCallback.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
Unhandled exception. System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'Microsoft.Win32.SafeHandles.SafeWaitHandle'.
   at Interop.Kernel32.SetEvent(SafeWaitHandle handle)
   at System.Threading.EventWaitHandle.Set()
   at Microsoft.Build.Execution.BuildManager.<>c__DisplayClass99_0.<BuildGraph>b__2(BuildSubmission finishedBuildSubmission)
   at Microsoft.Build.Shared.ThreadPoolExtensions.<>c__DisplayClass0_0.<QueueThreadPoolWorkItemWithCulture>b__0(Object state)
   at System.Threading.QueueUserWorkItemCallback.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
Unhandled exception. System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'Microsoft.Win32.SafeHandles.SafeWaitHandle'.
   at Interop.Kernel32.SetEvent(SafeWaitHandle handle)
   at System.Threading.EventWaitHandle.Set()
   at Microsoft.Build.Execution.BuildManager.<>c__DisplayClass99_0.<BuildGraph>b__2(BuildSubmission finishedBuildSubmission)
   at Microsoft.Build.Shared.ThreadPoolExtensions.<>c__DisplayClass0_0.<QueueThreadPoolWorkItemWithCulture>b__0(Object state)
   at System.Threading.QueueUserWorkItemCallback.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
Unhandled exception. System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'Microsoft.Win32.SafeHandles.SafeWaitHandle'.
   at Interop.Kernel32.SetEvent(SafeWaitHandle handle)
   at System.Threading.EventWaitHandle.Set()
   at Microsoft.Build.Execution.BuildManager.<>c__DisplayClass99_0.<BuildGraph>b__2(BuildSubmission finishedBuildSubmission)
   at Microsoft.Build.Shared.ThreadPoolExtensions.<>c__DisplayClass0_0.<QueueThreadPoolWorkItemWithCulture>b__0(Object state)
   at System.Threading.QueueUserWorkItemCallback.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
Unhandled exception. System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'Microsoft.Win32.SafeHandles.SafeWaitHandle'.
   at Interop.Kernel32.SetEvent(SafeWaitHandle handle)
   at System.Threading.EventWaitHandle.Set()
   at Microsoft.Build.Execution.BuildManager.<>c__DisplayClass99_0.<BuildGraph>b__2(BuildSubmission finishedBuildSubmission)
   at Microsoft.Build.Shared.ThreadPoolExtensions.<>c__DisplayClass0_0.<QueueThreadPoolWorkItemWithCulture>b__0(Object state)
   at System.Threading.QueueUserWorkItemCallback.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()

Steps to Reproduce

Build a specific internal repo

Expected Behavior

The build doesn't crash

Actual Behavior

The build crashes

Analysis

No response

Versions & Configurations

[dfederm]

May be related to #9983

My wild, outlandish, unsubstantiated guess is that the method BuildGraph is complete so the AutoResetEvent is being disposed, however it's being used from a separate thread (via innerBuildSubmission.ExecuteAsync) and throwing at the waitHandle.Set(); call.

[dfederm]

There is some evidence that this is a cascading failure for when a project cache plugin fails. If true, it's just a failure scenario that's not failing very gracefully.

[JanProvaznik]

Thanks for the report, could you please provide a reproduction for the bug or at least a .binlog file for a failing build?

[dfederm]

Looks like it may repro in other cases as well.

Either way though, looking at the code statically there is clearly a race condition there. The using should probably be removed there since it's used by another thread outside the scope of this function.

[JanProvaznik]

I agree there is a potential for a race condition. I am attempting to create a repro scenario to be sure a fix addresses it. Removing the using could help but the waitHandle should actually be disposed somewhere.

[adrianoc]

Hi, at Unity we are observing this error in our CI jobs from time to time; unfortunately we cannot update to version 17.13.9 (which includes the fix) because our build runs on .NET 8.

Is there any chance for this fix to be included in a version of the nuget package that supports .NET 8.0 ?

[JanProvaznik]

@adrianoc from looking at the code, the bug was introduced only for 17.11, so 17.10 and 17.8 which target .NET 8 should not run into this.

[adrianoc]

@JanProvaznik,

First of all, sorry for any confusion.

To give you more context:

Originally I though that that assembly was originating from a NuGet package, that's why I've asked about porting back, but it seems to me that the version from NuGet will only be used for compilation and that at runtime that assembly is resolved from the SDK folder, so updating the NuGet package would not solve my problem.

That said, looking into the IL of Microsoft.Build.dll for the .NET SDK 8.0.405 the code does have the using statement.

am I missing something ?

I also checked (8.0.408) and 9.x versions (I downloaded the binaries and looked into the IL) and AFAICT only .NET SDK 9.0.203 contains the fix.

[JanProvaznik]

@adrianoc .NET SDK 8.0.405, and 8.0408 corresponds to the 17.11 package, see table of the mapping .NET SDK 8.0.3xx and 8.0.1xx should not run into this bug. You can download these from here https://dotnet.microsoft.com/en-us/download/dotnet/8.0

[adrianoc]

Thanks @JanProvaznik, I tried version 8.0.312 and it does work but I am really confused here :)

Looking into this page, does that mean that SDK versions 8.0.409, 8.0.312 and 8.0.116:

1. targets the same runtime version (8.0.16)
2. the only difference between those SDKs is the version of MSBuild/Visual Studio they support

is that correct ?