swiftlang
Using recording an issue inside of a detached task causes a crash.
Description
This is something I learned while researching potential ways to address #475. Which, at the time I wrote that issue, I thought would have just resulted in the failure being silently dropped.
The current way of determining what the current reporting tests natively in Swift-Testing is broken when you run an expect inside of a detached task. Technically, you get a failing test result, but it's because the test runner crashed. I haven't yet spelunked enough through the codebase to figure out where that crash is.
This is because Test.current is stored as a task-local value. Which means that when you make an expect inside of a detached task, Swift Testing has lost that context, which means that Test.current is incorrectly (from the semantics of the current running test) as nil.
I don't know the concurrency system well enough to be able to suggest a good solution. The issue is that, because Swift Testing supports task-based parallelization, the value of Test.current strongly depends on the task it's running on. Which makes having a global Test.current like what's in swift-corelibs-xctest the wrong solution.
Expected behavior
No response
Actual behavior
No response
Steps to reproduce
This test should pass. The test runner crashes with a non-zero error instead.
import Testing
@Test func CheckingExpectWithDetachedTasks() async {
await Task.detached {
#expect(Bool(false))
}.value
}
swift-testing version/commit hash
Testing Library Version: 75.7 (arm64e-apple-macos13.0)
Swift & OS version (output of swift --version ; uname -a)
No response
The crash here is a known issue in Xcode 16 Beta, tracked by rdar://122190668. To resolve it, some work will be needed on the Xcode side, but it's also likely some additional action will be needed here in the swift-testing codebase, so I will leave this issue open to track that side of things.
Having not tried Swift Testing on Xcode 15, is the behavior that it silently drops the issue?
(To clarify: the crash is specific to Xcode 16.)
Right. But that said, other tools (such as VS Code or swift test CLI) may not be handling this situation very clearly in the reporting either, despite not crashing. Because of parallel execution, it's not always obvious which test a particular Issue is associated with. One idea I've had is to keep track of all the currently-running tests and include them as "potentially associated" tests in the reporting, so that at least you can narrow the scope down when troubleshooting.
To clarify more: the crash is specific to Xcode 16, which is the first version of Xcode to include support for Swift Testing. I meant that the crash is a bug in Xcode specifically, and not workflows that involve e.g. Linux or a package dependency on Swift Testing rather than Xcode and the copy of Swift Testing embedded in it.
I hope that makes more sense!
To clarify more: the crash is specific to Xcode 16, which is the first version of Xcode to include support for Swift Testing. I meant that the crash is a bug in Xcode specifically, and workflows that involve e.g. Linux or a package dependency on Swift Testing rather than Xcode and the copy of Swift Testing embedded in it.
I hope that makes more sense!
Oh! I gotcha! Thanks!
Folks, did a fix for this make it into Xcode 16.x yet? I'm seeing the same thing both locally and on CI:
XCTest/HarnessEventHandler.swift:282: Fatal error: Internal inconsistency: No test reporter for test case argumentIDs: Optional([Testing.Test.Case.Argument.ID(bytes: [34, 67, 52, 51, 56, 68, 66, 52, 65, 45, 57, 54, 70, 67, 45, 52, 70, 56, 66, 45, 66, 69, 48, 52, 45, 57, 53, 51, 56, 52, 69, 55, 53, 49, 49, 65, 57, 34])]) in test FormAssistantTests.PrefixedUniqueIdentifiableTests/testPrefixIsCorrectlyPrepended(to:)/PrefixedUniqueIdentifiableTests.swift:24:6
Interestingly, one project (SPM) I have was generally fine with Xcode 16.0 on macOS 15.0 but this is happening inconsistently (about 50% of the time) since 16.1/15.1 respectively.
I see there's an update in Xcode command-line tools coming down the pipe now (Oct 29 23:27 GMT), so will see if that makes a difference.
Xcode CL Tools don't include the component that causes the crash (as far as I'm aware, anyway.)
Xcode CL Tools don't include the component that causes the crash (as far as I'm aware, anyway.)
Yes, it made no difference. Didn't think it would; just wishful thinking.
This is a really frustrating bug as since 16.1 on 15.1 I'm having to run all my tests with swift test instead of ⌘U in Xcode.
I'm sorry it's not going well! Apple is tracking this issue, but since Xcode is not part of the open source Swift project, there's not a lot we can say publicly about it.
Hi guys. Is there any update about this? Is it possible to do a workaround to make the test pass?
For the moment, the workaround is to avoid using #expect() in detached tasks or external threads. If you need to use such an abstraction, use a continuation to suspend the test task until the detached work is complete, then call #expect() after the continuation resumes.
I'm pleased to report this bug in Xcode has been resolved as of Xcode 26.1 Beta. From the release notes:
Testing
Resolved Issues
- Fixed: If an issue is recorded during a Swift Testing test via an API such as
#expectorIssue.record()in a context which is unassociated with the test, such as viaTask.detached { … }or aDispatchQueue, the test process no longer unexpectedly terminates and Xcode shows the issue. (156631722)
If you continue to experience this issue after upgrading to that Beta Xcode version, please file a Feedback with Apple with details.
[!NOTE] Recording issues from contexts which are unassociated with a test, such as
Task.detached { ... }or aDispatchQueue, is still not recommended. In the reported results, such issues are represented as belonging to the test bundle or module, rather than a particular test, which can make them harder to diagnose. For best results, prefer recording issues from contexts which are associated with a test, such as the body of a@Testfunction or from non-detached, unstructured tasks whosevalueproperties areawait-ed on by a test function.