Problem to create tlb file on Windows 11 DevDrive

[swg08]

Hi, I'm evaluating DevDrive from Microsoft on a Windows 11. It works quite nicely, but we came across an issue that doesn't happen on non-DevDrives. (I replaced all real folder and filenames with %Folder% and %File%, hope it's not too confusing. Also the few words regarding the error output were in German - I translated them by hand to English)

%Folder%%File%.targets(158,5): error MSB3073: The comand ""%Folder%.nuget\packages\dspace.runtime.interopservices.buildtasks\1.11.0\tools\x64\dscom.exe" tlbexport "%Folder%%File%.dll" --silent --asmpath "C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App\9.0.1" --out "%Folder%%File%.tlb" --createmissingdependenttlbs false --embed "%Folder%%File%comhost.dll"" was terminated with Code 1 . [%Folder%%File%.csproj]

We have this happening on several machines that build our .NET solution. It happens exclusively on DevDrives. Might there be some issue with the ReFS file system, i.e. assumptions in the implementation that this file system might not guarantee or sporadically doesn't meet?

Are there known issues with Microsoft DevDrive or ReFS file system? Any ideas are welcome. How could we gain more information about the issue?

[SOsterbrink]

There are no known issues so far in connection with the dscom tooling and DevDrives. You are the first one to report it.

But I also have no experience with a DevDrive and don't use it on my machine. I will have to ask around if I can find someone who has experience with it.

[marklechtermann]

Thanks for your feedback,

can you run the test for dcom on your DevDrive? Maybe we'll see more then

git clone https://github.com/dspace-group/dscom.git
cd dscom
dotnet test

[DaveyJonesBitPail]

Entire output log (dotnet test): devdrive-test-execution.txt

Determining projects to restore... Restored V:\Repos\dscom\src\dscom.test.assembly\dscom.test.assembly.csproj (in 71 ms). Restored V:\Repos\dscom\src\dscom.test.assembly.dependency\dscom.test.assembly.dependency.csproj (in 69 ms). Restored V:\Repos\dscom\src\dscom.client\dscom.client.csproj (in 180 ms). Restored V:\Repos\dscom\src\dscom\dscom.csproj (in 180 ms). Restored V:\Repos\dscom\src\dscom.build\dscom.build.csproj (in 223 ms). Restored V:\Repos\dscom\src\dscom.test\dscom.test.csproj (in 228 ms). dscom.test.assembly.dependency -> V:\Repos\dscom\src\dscom.test.assembly.dependency\bin\Debug\net8.0\dSPACE.Runtime.InteropServices.Test.Assembly.Dependency.dll dscom.test.assembly.dependency -> V:\Repos\dscom\src\dscom.test.assembly.dependency\bin\Debug\net48\dSPACE.Runtime.InteropServices.Test.Assembly.Dependency.dll dscom.test.assembly.dependency -> V:\Repos\dscom\src\dscom.test.assembly.dependency\bin\Debug\net6.0\dSPACE.Runtime.InteropServices.Test.Assembly.Dependency.dll MinVer: Using { Commit: 2d0efe1, Tag: 'v1.15.1', Version: 1.15.1, Height: 0 }. MinVer: Calculated version 1.15.1. MinVer: Using { Commit: 2d0efe1, Tag: 'v1.15.1', Version: 1.15.1, Height: 0 }. MinVer: Calculated version 1.15.1. dscom -> V:\Repos\dscom\src\dscom\bin\Debug\net48\dSPACE.Runtime.InteropServices.dll dscom -> V:\Repos\dscom\src\dscom\bin\Debug\net8.0\dSPACE.Runtime.InteropServices.dll MinVer: Using { Commit: 2d0efe1, Tag: 'v1.15.1', Version: 1.15.1, Height: 0 }. MinVer: Calculated version 1.15.1. dscom -> V:\Repos\dscom\src\dscom\bin\Debug\net6.0\dSPACE.Runtime.InteropServices.dll dscom.test.assembly -> V:\Repos\dscom\src\dscom.test.assembly\bin\Debug\net8.0\dSPACE.Runtime.InteropServices.Test.Assembly.dll MinVer: Using { Commit: 2d0efe1, Tag: 'v1.15.1', Version: 1.15.1, Height: 0 }. MinVer: Calculated version 1.15.1. dscom.client -> V:\Repos\dscom\src\dscom.client\bin\Debug\net8.0\dscom.dll MinVer: Using { Commit: 2d0efe1, Tag: 'v1.15.1', Version: 1.15.1, Height: 0 }. MinVer: Calculated version 1.15.1. dscom.test.assembly -> V:\Repos\dscom\src\dscom.test.assembly\bin\Debug\net6.0\dSPACE.Runtime.InteropServices.Test.Assembly.dll MinVer: Using { Commit: 2d0efe1, Tag: 'v1.15.1', Version: 1.15.1, Height: 0 }. MinVer: Calculated version 1.15.1. dscom.build -> V:\Repos\dscom\src\dscom.build\bin\Debug\net6.0\dSPACE.Runtime.InteropServices.BuildTasks.dll dscom.build -> V:\Repos\dscom\src\dscom.build\bin\Debug\net8.0\dSPACE.Runtime.InteropServices.BuildTasks.dll dscom.test.assembly -> V:\Repos\dscom\src\dscom.test.assembly\bin\Debug\net48\dSPACE.Runtime.InteropServices.Test.Assembly.dll dscom.client -> V:\Repos\dscom\src\dscom.client\bin\Debug\net6.0\dscom.dll MinVer: Using { Commit: 2d0efe1, Tag: 'v1.15.1', Version: 1.15.1, Height: 0 }. MinVer: Calculated version 1.15.1. dscom.build -> V:\Repos\dscom\src\dscom.build\bin\Debug\net48\dSPACE.Runtime.InteropServices.BuildTasks.dll dscom.test -> V:\Repos\dscom\src\dscom.test\bin\Debug\net8.0\dscom.test.dll dscom.test -> V:\Repos\dscom\src\dscom.test\bin\Debug\net6.0\dscom.test.dll dscom.test -> V:\Repos\dscom\src\dscom.test\bin\Debug\net48\dscom.test.dll Test run for V:\Repos\dscom\src\dscom.test\bin\Debug\net8.0\dscom.test.dll (.NETCoreApp,Version=v8.0) VSTest version 17.11.1 (x64) Starting test execution, please wait... A total of 1 test files matched the specified pattern. Failed dSPACE.Runtime.InteropServices.Tests.CLITestEmbed.TlbEmbedAssemblyWithArbitraryIndex_ExitCodeIs0AndTlbIsEmbeddedAndValid [998 ms] Error Message: Assert.Equal() Failure: Values differ Expected: 0 Actual: 1 Stack Trace: at dSPACE.Runtime.InteropServices.Tests.CLITestEmbed.TlbEmbedAssemblyWithArbitraryIndex_ExitCodeIs0AndTlbIsEmbeddedAndValid() in V:\Repos\dscom\src\dscom.test\tests\CLITest\CLITestEmbed.cs:line 77 at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor) at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr) Failed dSPACE.Runtime.InteropServices.Tests.CLITestEmbed.TlbEmbedAssemblyWithArbitraryTlbAndArbitraryIndex_ExitCodeIs0AndTlbIsEmbeddedAndValid [767 ms] Error Message: Assert.Equal() Failure: Values differ Expected: 0 Actual: 1 Stack Trace: at dSPACE.Runtime.InteropServices.Tests.CLITestEmbed.TlbEmbedAssemblyWithArbitraryTlbAndArbitraryIndex_ExitCodeIs0AndTlbIsEmbeddedAndValid() in V:\Repos\dscom\src\dscom.test\tests\CLITest\CLITestEmbed.cs:line 93 at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor) at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr) Failed! - Failed: 2, Passed: 804, Skipped: 0, Total: 806, Duration: 5 s - dscom.test.dll (net8.0) Test run for V:\Repos\dscom\src\dscom.test\bin\Debug\net6.0\dscom.test.dll (.NETCoreApp,Version=v6.0) VSTest version 17.11.1 (x64) Starting test execution, please wait... A total of 1 test files matched the specified pattern. Failed dSPACE.Runtime.InteropServices.Tests.CLITestEmbed.TlbEmbedAssemblyWithArbitraryIndex_ExitCodeIs0AndTlbIsEmbeddedAndValid [802 ms] Error Message: Assert.Equal() Failure: Values differ Expected: 0 Actual: 1 Stack Trace: at dSPACE.Runtime.InteropServices.Tests.CLITestEmbed.TlbEmbedAssemblyWithArbitraryIndex_ExitCodeIs0AndTlbIsEmbeddedAndValid() in V:\Repos\dscom\src\dscom.test\tests\CLITest\CLITestEmbed.cs:line 77 at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor) at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr) Failed dSPACE.Runtime.InteropServices.Tests.CLITestEmbed.TlbEmbedAssemblyWithArbitraryTlbAndArbitraryIndex_ExitCodeIs0AndTlbIsEmbeddedAndValid [693 ms] Error Message: Assert.Equal() Failure: Values differ Expected: 0 Actual: 1 Stack Trace: at dSPACE.Runtime.InteropServices.Tests.CLITestEmbed.TlbEmbedAssemblyWithArbitraryTlbAndArbitraryIndex_ExitCodeIs0AndTlbIsEmbeddedAndValid() in V:\Repos\dscom\src\dscom.test\tests\CLITest\CLITestEmbed.cs:line 93 at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor) at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr) Failed! - Failed: 2, Passed: 804, Skipped: 0, Total: 806, Duration: 4 s - dscom.test.dll (net6.0) Test run for V:\Repos\dscom\src\dscom.test\bin\Debug\net48\dscom.test.dll (.NETFramework,Version=v4.8) VSTest version 17.11.1 (x64) Starting test execution, please wait... A total of 1 test files matched the specified pattern. Passed! - Failed: 0, Passed: 802, Skipped: 0, Total: 802, Duration: 5 s - dscom.test.dll (net48)

Visual Studio test explorer fail set:

As an example TlbEmbedAssemblyWithArbitraryTlbAndArbitraryIndex_ExitCodeIs0AndTlbIsEmbeddedAndValid will fail and result.StdErr will be:

Unhandled exception: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.ApplicationException: Error: Unable to write changes to assembly file 'V:\Repos\dscom\src\dscom.test.assembly\bin\Debug\net6.0\dSPACE.Runtime.InteropServices.Test.Assembly.comhost.dll' using TLB file 'V:\Repos\dscom\src\dscom.test.assembly\bin\Debug\net6.0\dSPACE.Runtime.InteropServices.Test.Assembly.tlb'; error code 5. at dSPACE.Runtime.InteropServices.TypeLibEmbedder.EmbedTypeLib(TypeLibEmbedderSettings settings) in V:\Repos\dscom\src\dscom\TypeLibEmbedder.cs:line 106 at dSPACE.Runtime.InteropServices.ConsoleApp.<>c.<ConfigureTLBEmbedHandler>b__4_0(TypeLibEmbedderSettings settings) in V:\Repos\dscom\src\dscom.client\Program.cs:line 166 at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor) at System.Reflection.MethodBaseInvoker.InvokeDirectByRefWithFewArgs(Object obj, Span1 copyOfArgs, BindingFlags invokeAttr) --- End of inner exception stack trace --- at System.Reflection.MethodBaseInvoker.InvokeDirectByRefWithFewArgs(Object obj, Span1 copyOfArgs, BindingFlags invokeAttr) at System.Reflection.MethodBaseInvoker.InvokeWithOneArg(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture) at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture) at System.Delegate.DynamicInvokeImpl(Object[] args) at System.CommandLine.NamingConventionBinder.ModelBindingCommandHandler.InvokeAsync(InvocationContext context) at System.CommandLine.NamingConventionBinder.ModelBindingCommandHandler.Invoke(InvocationContext context) at System.CommandLine.Invocation.InvocationPipeline.<>c__DisplayClass4_0.<<BuildInvocationChain>b__0>d.MoveNext()--- End of stack trace from previous location --- at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass17_0.<<UseParseErrorReporting>b__0>d.MoveNext()--- End of stack trace from previous location --- at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass12_0.<<UseHelp>b__0>d.MoveNext()--- End of stack trace from previous location --- at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass22_0.<<UseVersionOption>b__0>d.MoveNext()--- End of stack trace from previous location --- at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass19_0.<<UseTypoCorrections>b__0>d.MoveNext()--- End of stack trace from previous location --- at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<UseSuggestDirective>b__18_0>d.MoveNext()--- End of stack trace from previous location --- at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass16_0.<<UseParseDirective>b__0>d.MoveNext()--- End of stack trace from previous location --- at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<RegisterWithDotnetSuggest>b__5_0>d.MoveNext()--- End of stack trace from previous location --- at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass8_0.<<UseExceptionHandler>b__0>d.MoveNext()

[DaveyJonesBitPail]

A couple of additional notes:

• Issue appears to always occur at TypeLibEmbedder.cs#L103-L107
  • Specifically actually applying the internal changeset w/ EndUpdateResourceW will result in ERROR_ACCESS_DENIED (Error code 5)
    • DevDrives disable the usage of Minifilters for their volumes so I can't capture the File operations via a tool like Procmon.
    • I haven't delved deeply into ETW events to try to ascertain what specific process is causing this (DevDrive-specific, ReFs-specific, Anti-Virus-specific, etc)
• Anecdotally: Re-trying just the EndUpdateResourceW call will always fail
• Re-composing and attempting to apply the resource changeset (new calls to BeginUpdateResource, UpdateResource, EndUpdateResource) usually works.

Here is a ham-fisted attempt to try to treat the symptoms (rather than the cause which I haven't been able to really isolate) by allowing embed retries: dscom-fork/staging-pr-prep specifically TypeLibEmbedder changes. This works on our end so far, but I am not able to confirm whether this is treating something specific to our environment or whether it is common to Win 11 + DevDrive usage.

[marklechtermann]

I attempted to reproduce the error but was unable to do so. Here are the steps I followed:

1. Launched a Windows 11 Pro 24H2 machine in Azure (Standard D8s v4 (8 vcpus, 32 GiB memory).
2. Installed .NET 8.0, Git, and VSCode.
3. Created a new DevDrive with ReFS as the file system (D:).
4. Cloned the repository dspace-group/dscom to d:\git\dscom.
5. Both dotnet test and the tests in VSCode ran without any errors.

Try reproducing the setup on a virtual machine in Azure or AWS. Could this be an issue with your antivirus software? If implementing a retry resolves the problem, it might indicate that the antivirus is interfering.

[DaveyJonesBitPail]

I attempted to reproduce with a personal machine (Win 11 Pro v10.0.22631 Build 22631) on a Dev Drive (ReFS, partition) and had the same results you posted (All green, no failures) across multiple attempts.

I am at least quite convinced that the issue is something specific to our environment, specifically our antivirus.

As it stands now we have "made do" with the fork I mentioned earlier that allows for the possibility of configuring retry on failure for embedding tlbs. Do you believe there is any value in adding a PR to this repo with those changes in the slim chance another individual has such problems and is not able to modify their IT env?

[marklechtermann]

I believe this change is too specific to your particular use case and may not be beneficial for the broader user base. Implementing a retry mechanism in the main branch could mask underlying issues that should ideally be addressed directly within your IT environment. The retry mechanism does not address the root cause of the problem, which is the antivirus software blocking the file. It is generally better to resolve such issues at their source. Introducing retries adds complexity to the codebase, making it harder to maintain and debug in the future.

I recommend addressing this issue locally with a batch script (if you use the dscom.exe directly from this batch file, and do not use the build task) that handles the retries on your end. This approach keeps the main codebase clean and ensures that the solution is tailored to your specific environment.

For example like this:

@echo off
set RETRY_COUNT=3
set COUNT=0

:retry
set /a COUNT+=1
echo Attempt %COUNT% of %RETRY_COUNT%

dscom.exe [some options .......]

if %ERRORLEVEL% neq 0 (
    if %COUNT% lss %RETRY_COUNT% (
        echo Error encountered. Retrying in 3 seconds...
        timeout /t 3 /nobreak >nul
        goto retry
    ) else (
        echo Failed after %RETRY_COUNT% attempts.
        exit /b 1
    )
) else (

.... 
    

Or what do you think @SOsterbrink @matthiasnissen ?

[marklechtermann]

I would like to close the issue. Please let me know if there are any outstanding concerns or if we can proceed with closing it.

[SOsterbrink]

I agree with you that adding a retry functionality to dscom would introduce complexity and error potential which is much easier to handle with external scripts. In my opinion we can close it because the error is not causes by dscom.