spek icon indicating copy to clipboard operation
spek copied to clipboard

CancellationExceptions cancel tests successfully rather than failing them

Open ssp opened this issue 5 years ago • 1 comments

A CancellationException being thrown in in the tested code does not result in a failing test.

In particular this means that tests running into timeouts with Kotlin’s withTimeout() {} will succeed as this throws a TimeoutCancellationException.

To me that is unexpected and led to tests which had broken succeeding nonetheless.

Example:

package integrationtest

import kotlinx.coroutines.CancellationException
import kotlinx.coroutines.delay
import kotlinx.coroutines.runBlocking
import kotlinx.coroutines.withTimeout
import org.spekframework.spek2.Spek
import org.spekframework.spek2.style.specification.describe

object CancellationTest : Spek({
    describe("Cancellation Exception does not fail test") {

        it("Superclass RuntimeException makes test fail") {
            throw RuntimeException()
        }

        it("CancellationException does not fail test") {
            throw CancellationException()
        }

        it("timeout does not make test fail") {
            runBlocking {
                withTimeout(1) {
                    delay(10)
                }
            }
        }
    }
})

ssp avatar Jun 29 '20 07:06 ssp

Thanks for the report @ssp! Nothing I can do at the moment but wait until the coroutine support is added (which is coming in 2.1 - no release dates yet as the release is blocked by a problem with kotlin compiler plugins).

raniejade avatar Jun 29 '20 11:06 raniejade