QuicTestKeyUpdate test sometimes fail

[ami-GS]

Describe the bug

Sometimes QuicTestKeyUpdate test fail https://dev.azure.com/ms/msquic/_build/results?buildId=431415&view=logs&j=3667e279-20cb-5b66-4f7d-50a166799211&t=69945233-3564-536d-0950-ae97d4372f33&l=1480

maybe windows only?

Affected OS

• [X] Windows
• [ ] Linux
• [ ] macOS
• [ ] Other (specify below)

Additional OS information

No response

MsQuic version

main

Steps taken to reproduce bug

.\artifacts\bin\windows\x64_Debug_schannel\msquictest.exe --gtest_filter="Misc/WithKeyUpdateArgs1.KeyUpdate/*"

Expected behavior

Success

Actual outcome

Sometimes Failed (feels 0.0001% level on local machine, more frequently on CI)

Additional details

No response

[ami-GS]

Success.log Fail.log

[ami-GS]

Server->ForceKeyUpdate() failed by QUIC_STATUS_INVALID_STATE Then second Client->ForceKeyUpdate() continue failing by QUIC_STATUS_INVALID_STATE

Trying to repro to know which connection condition matches

[ami-GS]

https://github.com/microsoft/msquic/blob/1edd42421a106d35381f4e0a93605860292026c0/src/test/lib/HandshakeTest.cpp#L2203-L2209 --- Expected case

1. Client executes ForceKeyUpdate (SetParam QUIC_PARAM_CONN_FORCE_KEY_UPDATE)
2. Server executes ForceKeyUpdate (SetParam QUIC_PARAM_CONN_FORCE_KEY_UPDATE)

--- problematic case

1. Client ForceKeyUpdate
2. Client sends packet with key type of QUIC_PACKET_KEY_1_RTT_NEW during 1.
3. Server receives packet and update key phase https://github.com/microsoft/msquic/blob/1edd42421a106d35381f4e0a93605860292026c0/src/core/connection.c#L4338-L4351
4. Server changes Connection->Packets[QUIC_ENCRYPT_LEVEL_1_RTT]->AwaitingKeyPhaseConfirmation = TRUE;
5. Server executes ForceKeyUpdate, but failed as QUIC_STATUS_INVALID_STATE due to AwaitingKeyPhaseConfirmation