Extending TAP to define parameters controlling the TAP producers.

[0xorial]

Hi,

I am looking at TAP from the point of view of IDE integration. TAP currently defines a way for test runners to report testing results to consumers, hence IDE being a consumer can reuse the reporting parsing code, while using test runners interchangeably. However, there are 2 other common features that IDEs implement:

• enumerating all tests without running them (happens when user opens workspace)
• running selected subset of tests(happens when user selects a single test and runs)

Currently IDEs would have to implement this logic for each test runner, even if the runner itself is a TAP producer. This makes it impossible to make a universal TAP test adapter for an IDE and just use TAP-compatible runners interchangeably.

My idea is currently that TAP can have a small extension, defining 2 CLI parameters:

• --dry-run to produce the TAP output just skipping every test.
• --tests only run tests whose description matches the provided glob.

Please let me know if what I am describing makes sense.

[exodist]

The idea of TAP is a universal output, or result. The idea is that any "TAP consumer" can parse what a "TAP produce" spits out. Defining CLI arguments for producers is out of scope. I think this is a step that would be a mistake to take. This is additionally true if we consider existing TAP producers, which are usually harnesses that usually already have --dry-run or similar flags, and may or may not already implement --tests.

We do not want to start enforcing arguments on harnesses/producers

[isaacs]

Agreed. Out of scope for the protocol specification, this type of behavior is left for producers.