Windows-Containers Kerberos broken in recent nanoserver images

Describe the bug It looks like nanoserver-ltsc2022 images released in April 2024 and later have broken Kerberos.

To Reproduce First, create a simple ASP.NET Core project AspNetKerbHello.

Create AspNetKerbHello.csproj with the following content:

<Project Sdk="Microsoft.NET.Sdk.Web">

  <PropertyGroup>
    <TargetFramework>net8.0-windows</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.AspNetCore.Authentication.Negotiate" Version="8.0.8" />
  </ItemGroup>

</Project>

Now, create Program.cs with the following content - we are enabling negotiate authentication here:

using Microsoft.AspNetCore.Authentication.Negotiate;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddAuthentication(NegotiateDefaults.AuthenticationScheme)
    .AddNegotiate();

builder.Services.AddAuthorization(options =>
{
    options.FallbackPolicy = options.DefaultPolicy;
});

var app = builder.Build();

app.UseAuthentication();
app.UseAuthorization();

app.MapGet("/hello", (HttpContext context) =>
{
    var userName = context.User.Identity?.Name ?? "Anonymous";
    var authType = context.User.Identity?.AuthenticationType ?? "None";
    return Results.Ok($"Hello, {userName}! - via {authType}");
});

app.Run();

Now build the above project, using dotnet build command.

Now, inside the build output directory where AspNetKerbHello.dll is present, create a Dockerfile with the following content. We are purposefully pulling .NET runtime 8.0.4 because it uses nanoserver image from April 2024 - i.e. mcr.microsoft.com/windows/nanoserver:ltsc2022-KB5036909 -- but this works with even the latest nanoserver image:

FROM mcr.microsoft.com/dotnet/aspnet:8.0.4-nanoserver-ltsc2022
WORKDIR /app
EXPOSE 80
ENV ASPNETCORE_URLS=http://+:80

COPY . .

USER ContainerUser
ENTRYPOINT ["dotnet", "AspNetKerbHello.dll"]

Now build this docker image, let's assume it's tagged aspnetkerbhello:v1:

docker build -t aspnetkerbhello:v1 .

Now run this image in Kubernetes as a gMSA that owns a SPN

Now, access the Kubernetes SVC URI - assuming the SPN owned by the gMSA is http/<FQDN> where FQDN is Kubernetes SVC domain name. In the below example SPN is http/my-svc.my-namespace.svc.cluster.local:

Invoke-WebRequest -UseBasicParsing -UseDefaultCredentials -AllowUnencryptedAuthentication "http://my-svc.my-namespace.svc.cluster.local/hello"

Assuming NTLM is disabled and authentication is happening over Kerberos, you will get HTTP 500 error with an exception containing the following error callstack in the Kubernetes pod logs:

fail: Microsoft.AspNetCore.Authentication.Negotiate.NegotiateHandler[5]
      An exception occurred while processing the authentication request.
      System.Net.InternalException: Exception of type 'System.Net.InternalException' was thrown. -1073741428
         at System.Net.SecurityStatusAdapterPal.GetSecurityStatusPalFromInterop(SECURITY_STATUS win32SecurityStatus, Boolean attachException)
         at System.Net.NegotiateAuthenticationPal.WindowsNegotiateAuthenticationPal.AcceptSecurityContext(SafeFreeCredentials credentialsHandle, SafeDeleteContext& securityContext, ContextFlags requestedContextFlags, ReadOnlySpan`1 incomingBlob, ChannelBinding channelBinding, Byte[]& resultBlob, Int32& resultBlobLength, ContextFlags& contextFlags)
         at System.Net.NegotiateAuthenticationPal.WindowsNegotiateAuthenticationPal.GetOutgoingBlob(ReadOnlySpan`1 incomingBlob, NegotiateAuthenticationStatusCode& statusCode)
         at System.Net.Security.NegotiateAuthentication.GetOutgoingBlob(ReadOnlySpan`1 incomingBlob, NegotiateAuthenticationStatusCode& statusCode)
         at System.Net.Security.NegotiateAuthentication.GetOutgoingBlob(String incomingBlob, NegotiateAuthenticationStatusCode& statusCode)
         at Microsoft.AspNetCore.Authentication.Negotiate.NegotiateState.GetOutgoingBlob(String incomingBlob, BlobErrorType& status, Exception& error)
         at Microsoft.AspNetCore.Authentication.Negotiate.NegotiateHandler.HandleRequestAsync()

If you made a mistake somewhere, you will instead get authenticated over NTLM and won't see the error - rather a 200 OK page with the following message:

"Hello, contoso\\username! - via NTLM"

The issue only occurs when using Kerberos, not NTLM.

Now, all of the above can be also done without involving Kubernetes with a docker container running with gMSA cred-spec file but it's bit tricky - you need to first enable Kerberos on loopback using DisableLoopbackCheck registry key, and then use [System.Net.AuthenticationManager]::CustomTargetNameDictionary to map localhost to the SPN owned by the gMSA; and even then you might endup with NTLM. So I recommend using Kubernetes for testing the above.

You do not need multiple replicas. Even just a single pod runs into the above error.

I know the issue isn't because of any change in the .NET runtime version because I have created a custom .NET runtime image with the exact same runtime version (8.0.4) but older (March 2024) nanoserver base image and I couldn't repro the issue - suggesting the issue is in the nanoserver base image.

Expected behavior

Kerberos should work with nanoserver images

Configuration:

Edition: Windows Server 2022 with September 2024 build
Base Image being used: nanoserver-ltsc2022
Container engine: docker and containerd both
Container Engine version: docker 26 / containerd 1.6.31

Additional context

Per https://github.com/dotnet/runtime/discussions/105567#discussion-6980650 the error System.Net.InternalException: Exception of type 'System.Net.InternalException' was thrown. -1073741428 translates to ERROR_TRUSTED_DOMAIN_FAILURE or The trust relationship between the primary domain and the trusted domain failed, so I suspect this is likely related to the other gMSA issue https://github.com/microsoft/Windows-Containers/issues/405

Sep 15 '24 20:09 avin3sh

Thank you for creating an Issue. Please note that GitHub is not an official channel for Microsoft support requests. To create an official support request, please open a ticket here. Microsoft and the GitHub Community strive to provide a best effort in answering questions and supporting Issues on GitHub.

Sep 15 '24 20:09 github-actions[bot]

I'm trying to reproduce but the above command " Invoke-WebRequest -UseBasicParsing -UseDefaultCredentials -AllowUnencryptedAuthentication "http:57.151.34.124/hello" Invoke-WebRequest : A parameter cannot be found that matches parameter name 'AllowUnencryptedAuthentication'." @avin3sh, can you please confirm this is the command you initiated from client side? thanks

Oct 16 '24 23:10 zylxjtu

hi @zylxjtu - could you please try PowerShell Core/7.1. I don't know what's equivalent of AllowUnencryptedAuthentication in Windows PowerShell.

Oct 18 '24 18:10 avin3sh

No updates on this yet from the internal team. (55132872)

Dec 10 '24 19:12 ntrappe-msft

This will be treated as part of the original Issue.

Jan 28 '25 19:01 ntrappe-msft

Thanks for the update, @ntrappe-msft. For the avoidance of any doubt - the above issue exists even with the gMSA Webhook based workaround proposed in #405 - UNLESS we change the container user to NT Authority\NetworkService instead of ContainerUser.

Jan 29 '25 08:01 avin3sh