Data Driven Specs

[leonard84]

It is an often requested feature and with the new flexibility that the JUnit Platform gives us, we can finally implement it in a nice way. (This issue supersedes #668).

Goals

• Offer a way to execute a whole Specification for a set of parameters, similar to data-driven features.
• A data-driven Specification should behave like a normal one, with the addition that there are injected parameters.
• This feature should integrate well with the existing functionality in terms of UX/DX. It should feel natural.

Open Questions

1. How do we deal with inheritance?
   1. Forbid Inheritance
   2. Create the Cartesian product of the parameters
   3. Treat them as extensions, requiring same size and just join both parameters
2. How do we define the data-variables?
   1. with a special static where method
   2. with a static inner class that is either annotated or implements a certain marker interface and has a where method or something similar
3. How do we inject the data-variables?
   1. automatically define them as properties with the same name
   2. require the user to define the properties in the spec, allowing typing
   3. a combination of the above
   4. if we use a the inner class approach (2.ii) we can just inject a instance of that class and have the properties be members of that class, either explicit or implicit. This would have the added benefit of isolating them visually in the code.
4. How do we deal with spring or similar extensions
   1. the spring extension can currently only inject in non-shared fields (there are a workarounds for that https://github.com/spockframework/spock/issues/76#issuecomment-241698357). However, those will probably not work for the spec level.
5. How do we deal with @Shared and setupSpec/cleanupSpec for Spec iterations
   1. Add some kind of @SharedForIteration and setupSpecIteration/cleanupSpecIteration
   2. don't support iteration specific sharing
   3. something else
6. How to report during discovery (currently, data-driven features are reported purely dynamic, if we did the same the specs then they wouldn't report any features during discovery)
   1. Don't report any features during discovery
   2. Distinguish between data-driven and non-data-driven specs and report feature for the latter
   3. Somehow detect self contained data providers (i.e., ones without field or function access) and run them during discovery, (this could be ported to features as well)
   4. something else

Please comment: @spockframework/supporter

[Vampire]

re. 1.: Not sure yet what I like most or feels best Other options or detail could be to have the where method in the sub-spec, that injects data to a property defined in the super-spec, or to not allow a where method in the sub-spec but still allow a sub-spec, so that the super-spec defines the spec-iterations, while the sub-spec could then provide features to be executed about the spec-iterations. But I think I maybe most tend to cartesian product, so that you can e. g. have a base-spec that creates spec iterations for various Gradle verison and Java version combinations and still can have other additional factors in sub-specs, when for example testing a Gradle plugin.

re. 2.: I think I'd vote for i.a., that is special where method the content of which follows the same rules and possibilities as the where block in a feature, but not static. setupSpec and cleanupSpec are not static either. Having the where method besides the feature methods for me feels most consistent with having the where block besides the other blocks and thus feels most spocky.

re. 3.: iii. feels most easy and consistent with data-driven features. The parameters of a feature are like the properties of the spec and in line with the PRs of me just merged, I would also like to see the same bahviour here, defining missing properties automatically while using already defined properties with their type, even when we here probably cannot fail if something defined got no value.

re. 4.: no idea

re. 5.: i. or iii. the question is, what the meaning of @Shared will be. I'm not sure what should be considered closer to the current behavior, shared within the spec iteration and there will something new for sharing across iterations, or shared across iterations and a new annotation for shared within the iteration.

[marcphilipp]

re. 6.: I'd go with ii. but for consistency with data-driven feature methods, I'd report an additional container node that will become the parent of all spec nodes of the data-driven spec at execution time.

[rieske]

Gradle has a few use cases for such feature that were previously implemented in JUnit4's Runner infrastructure and now I have attempted migrating them to Spock2 and to use extensions/interceptors for that: https://github.com/gradle/gradle/pull/15710 specifics: https://github.com/gradle/gradle/pull/15710/files#diff-84b2d9bb72ca22865aa56bbae9b5412bbc5e863f14ca861a36398faadf06c114R39 My current attempt has one specific shortcoming - the test runs for each "data" iteration are nothing but a for loop that performs some data-setting action before running the intercepted test method and then cleaning up after the test method. This does not make the testing framework see those "data" iterations as separate tests.

Use cases we are trying to solve:

1. Tests/Specs specify a pattern that can be parsed/interpreted by user code (as opposed to framework code) that then sets up the spec instance before running its setup methods and features. The pattern that is parsed can yield multiple different variants that then require the spec/feature to be executed multiple times with different setups.

For example:

// At the moment we use custom annotations to capture the patterns for the data feed.
// They can be per-spec and per-feature. Subclass annotations win over superclass annotations. Feature annotations win over spec annotations.
@DataIterations("1 to 3") 
class SomeSpec extends Specification {
    def 'some feature'() { /* ... */ }
    def 'some feature with where clause'() { /* ... where: foo << ['a', 'b'] */ }
    // ...
}

Our code (at the moment hooked using extension mechanism) parses the @DataIterations annotation and determines that it needs to run SomeSpec with some data, derived from pattern "1 to 3". It could be to load contents of files 1, 2 and 3 to some spec instance variables of the spec to be used in each iteration. Or it could be to load different versions of some tool and set some spec instance variables that the tests use. Then, such spec would be executed 3 times:

SomeSpec
|--some feature[1]
|--some feature[2]
|--some feature[3]
|--some feature with where clause[1]
    |--a
    |--b
|--some feature with where clause[2]
    |--a
    |--b
|--some feature with where clause[3]
    |--a
    |--b

or

SomeSpec[1]
|--some feature
|--some feature with where clause
    |--a
    |--b
SomeSpec[2]
|--some feature
|--some feature with where clause
    |--a
    |--b
SomeSpec[3]
|--some feature
|--some feature with where clause
    |--a
    |--b

Currently, using the existing interceptor mechanism we achieve something like:

SomeSpec
|--some feature[1, 2, 3] // intercept feature; for i in 1,2,3 do: setup spec with i; run feature; cleanup spec; done
|--some feature with where clause
    |--a [1, 2, 3] // intercept iteration; for i in 1,2,3 do: setup spec with i; run iteration; cleanup spec; done
    |--b [1, 2, 3] // intercept iteration; for i in 1,2,3 do: setup spec with i; run iteration; cleanup spec; done

2. A similar one - I want to run the spec against my product with some feature turned on/off, or some feature flag set using a range of values - user code knows how many variants of the Spec to create. Here, similarly we intercept the feature/iteration execution and configure some instance variable before running the test method.

3. We also have a use case where we want to load the spec class using different classloaders for different executions. In JUnit platform, the test classes are loaded during test discovery phase - at the moment we can achieve what we need at JUnit engine level, but perhaps this could be something that Spock could expose a hook for as well.