standarize/introduce Description links

[piotrturski]

currently the only way to interact with IDE is a Description. based on the Description IDE has to create nice looking tree and navigate to source when user clicks on this tree.

there are more and more custom Runners that introduces new things (parameters, nested tests etc). but still there is no standard, portable way to link to correct method. it would be good to separate what's displayed in the tree from what method that node is linked to. then intermediate nodes in parameter tests could be named as user/library creator wants yet still after double-click they will link to correct methods. same with leaf nodes

[kcooney]

Do you have any specific recommendations?

Edit: also would be good to have examples of open source runners that would benefit from this proposal (preferably, ones that are actively maintained)

[piotrturski]

nothing detailed. i would leave fDisplayName and stop adding there full class name and introduce new field link. that should be able to link to classes and method. maybe even it should link to specigic parameter (in this case we need source line and row number) so i'm not sure if it should be a string or something more complex. when IDEs start using it, the navigation will be much more procise and comfortable

[kcooney]

It sounds like the challenge here is to work with the IDEs to understand what they would need from the "link" to do what you want. Could you reach out to the Eclipse or IntelliJ developers and come up with a specific proposal?

[marcphilipp]

Interesting, I've also been thinking about this.

We could add some kind of marker interface, e.g. TestLocation, along with a few general purpose implementations:

• ClassLocation: a Java Class
• MethodLocation: a Java Method (and Class?)
• FileLocation: a File object along with an optional line number
• UrlLocation: a standard URL or URI object

Thoughts?

[jlink]

Where would the information, e.g. file and line number, come from? I know that the groovy compiler adds all that stuff to the AST. Is there anything like that available in Java via reflection or alike? My impression was that IDEs add those information themselves since they usually will parse all sources.

But maybe I misunderstood you and FileLocation would only be used for file-based tests by external runners.

Johannes

2015-01-19 19:13 GMT+01:00 Marc Philipp [email protected]:

Interesting, I've also been thinking about this.

We could add some kind of marker interface, e.g. TestLocation, along with a few general purpose implementations:

• ClassLocation: a Java Class
• MethodLocation: a Java Method (and Class?)
• FileLocation: a File object along with an optional line number)
• UrlLocation: a standard URL or URI object

Thoughts?

— Reply to this email directly or view it on GitHub https://github.com/junit-team/junit/issues/1070#issuecomment-70537054.

[piotrturski]

@kcooney i add posts on both intellij and eclipse devs forum. don't know if there will be any reposnose. probably we have to wait.

@jlink no, it's not IDE that adds it. when you thow and exception this informartion is in the stacktrace. it's added by the compiler. although during compilation you can explicitly request to remove those information almost noone does that

@marcphilipp currently IDEs do it differently by parsing Description. for example intellij doesn't navigate to methods with parameters. eclipse is a bit better here. for sure MethodLocation (with parameters) will be better than what is done currently. but in complex scenarios we have at least two lacations: test mathod and parameters for this specific invocation. mathod is in A.java and parameters are in data.txt row 50 column 16. sometimes both are in same file (@Parameterized, zohhak, junitParams, junit-dataprovider etc) but in different lines. and remember that there can be many different parameters set in the same line - so column is also important. And sometimes it's not important - if we generate parameters on the fly

[kcooney]

Note that the Description is created by the Runner, usually before the test is run, and is essentially immutable (in JUnit 5, we hope to make it immutable).

@piotrturski can you add links to the Eclipse and IntelliJ discussions here?

[jlink]

2015-01-19 22:10 GMT+01:00 Piotr Turski [email protected]:

@jlink https://github.com/jlink no, it's not IDE that adds it. when you thow and exception this informartion is in the stacktrace. it's added by the compiler. although during compilation you can explicitly request to remove those information almost noone does that

So this only works after a test failure? How do you link test cases without failures to their source?

Johannes

2015-01-19 22:10 GMT+01:00 Piotr Turski [email protected]:

@kcooney https://github.com/kcooney i add posts on both intellij and eclipse devs forum. don't know if there will be any reposnose. probably we have to wait.

@jlink https://github.com/jlink no, it's not IDE that adds it. when you thow and exception this informartion is in the stacktrace. it's added by the compiler. although during compilation you can explicitly request to remove those information almost noone does that

@marcphilipp https://github.com/marcphilipp currently IDEs do it differently by parsing Description. for example intellij doesn't navigate to methods with parameters. eclipse is a bit better here. for sure MethodLocation (with parameters) will be better than what is done currently. but in complex scenarios we have at least two lacations: test mathod and parameters for this specific invocation. mathod is in A.java and parameters are in data.txt row 50 column 16. sometimes both are in same file (@parameterized https://github.com/parameterized, zohhak, junitParams, junit-dataprovider etc) but in different lines. and remember that there can be many different parameters set in the same line - so column is also important. And sometimes it's not important - if we generate parameters on the fly

— Reply to this email directly or view it on GitHub https://github.com/junit-team/junit/issues/1070#issuecomment-70561355.

[piotrturski]

this information is always available: http://stackoverflow.com/questions/7483421/how-to-get-source-file-name-line-number-from-a-java-lang-class-object

but after some reading it seems like the resolution is not goof enough :( there is probably no way to get column number and line number may also be a bit inaccurate. so i think all can be done is just better method linking, for example adding full signature instead of only name

[kcooney]

@piotrturski generally test methods don't have parameters, so the method name is sufficient. Is this specially for the case where the runner allows test methods to have parameters? If so, what information besides the signature would you need, and what will the caller do with this data?

In other words, can you provide a specific use case?

[jlink]

2015-01-20 16:35 GMT+01:00 Piotr Turski [email protected]:

this information is always available:

http://stackoverflow.com/questions/7483421/how-to-get-source-file-name-line-number-from-a-java-lang-class-object

but after some reading it seems like the resolution is not goof enough :( there is probably no way to get column number and line number may also be a bit inaccurate. so i think all can be done is just better method linking, for example adding full signature instead of only name

One could create file and line info for assertions (failed and succeeded ones). For test classes, methods and alike one could link to the most specific java.lang.reflect-Object which would probably help the IDE to find out about file and line number.

Johannes

[piotrturski]

@kcooney specific case is method with parameters. i'm aware that default runner uses methods without parameters. but if junit supports custom runners then better API for IDEs would be nice

[kcooney]

@piotrturski so for the case of methods with parameters, specifically what data would want that we don't provide in Description today?

Other than methods for parameters, what would you need from description "links"?

Note that if the intention is to use the data from the IDEs, then we would really need to work with the IDEs to design this. If we design in a vacuum the API will likely not be usable to the IDE teams.

Note I can imagine cases where the data passed a parameterized test could not be known until right before the test is run. Perhaps the data you want shouldn't be attached to the Description but instead added to the Failure or some other object that we pass to the listener.

[piotrturski]

in order to navigate to a method with parameters we need signatures. easiest way is to use JNI type signatures like (ILjava/lang/String;[I). details here: http://docs.oracle.com/javase/1.5.0/docs/guide/jni/spec/types.html at the beginning i was thinking about simply keeping a reference to Method but IDEs may keep it in different classloaders or even different jvms. for the same reason, re-running tests with its parameter may be possible only using serialization (i don't have other ideas at the moment) but requiring users to make their parameters serializable is unnecessary overkil and limitation

[kcooney]

@piotrturski IDEs run tests in different processes, which is why Description is serializable.

Any luck reaching out to the IDE developers?

[piotrturski]

nope. i posted on their forums and link to this discussion. so probably noone is interested

[kcooney]

@piotrturski sorry to hear that. Part of the problem may be that the description in the bugs are a bit vague (I couldn't find the Eclipse bug, but I did eventually find the IntelliJ bug).

I suggest:

1. Changing the name of this feature request to be more specfiic
2. Describe what your immediate problem is. Don't try to overgeneralize yet. If other people have similar problems, we can try to generalize
3. If possible, provide a link to the code of your custom runner
4. Describe the current result in JUnit and your IDE
5. Suggest a change we can make in JUnit that can help with your problem
6. Suggest how the IDE would take advantage of that change to fix your problem

[akozlova]

Hi guys,

I am responsible for junit integration in IntelliJ, sorry that I didn't saw the forum post earlier.

The 'link' which describes the class fully qualified name and the method signature (JVM signature is ok) would help a lot. Now this information is supported based on listener in the test's VM: description.getMethodName(), description.getClassName() and description.getDisplayName() for old versions are used. This way we don't have e.g. class name when junit 3 named test suite is run, no navigation there is provided. I don't remember any other cases where navigation with standard runners is impossible. Custom runners on the other hand are not supported.

If you are interested in details I can explain how it works in IDEA further (with github links?). If you know any other problems with IDEA, please notify me (e.g. http://youtrack.jetbrains.com).

Thank you for your attention to our problems! Anna

[jlink]

Hi Anna,

More details would be greatly appreciated since we (the JUnit team) are currently thinking about ways to facilitate/standardize IDE and tools integration as much as possible. Maybe you'd even be interested in collaborating about the "perfect" integration API?

cheers, Johannes

2015-01-26 10:52 GMT+01:00 akozlova [email protected]:

Hi guys,

I am responsible for junit integration in IntelliJ, sorry that I didn't saw the forum post earlier.

The 'link' which describes the class fully qualified name and the method signature (JVM signature is ok) would help a lot. Now this information is supported based on listener in the test's VM: description.getMethodName(), description.getClassName() and description.getDisplayName() for old versions are used. This way we don't have e.g. class name when junit 3 named test suite is run, no navigation there is provided. I don't remember any other cases where navigation with standard runners is impossible. Custom runners on the other hand are not supported.

If you are interested in details I can explain how it works in IDEA further (with github links?). If you know any other problems with IDEA, please notify me (e.g. http://youtrack.jetbrains.com).

Thank you for your attention to our problems! Anna

— Reply to this email directly or view it on GitHub https://github.com/junit-team/junit/issues/1070#issuecomment-71436462.

[akozlova]

Ok, let me explain.

When the user has tests under selection, IDEA suggests to run all tests in this package, in class, test method, etc. When the configuration is run, IDEA writes file with description which tests should be run, and starts another VM where junit is started (JUnit4IdeaTestRunner) with the listener which sends IDEA notifications (JUnit4TestResultsSender). JUnit4OutputObjectRegistry#addStringRepresentation describes how test is passed to IDEA's VM, actually method name and class name are serialized (JUnit4ReflectionUtil#getMethodName). IDEA on its side finds the class by fully qualified name and then test method inside it. Test tree consists of TestProxy nodes. Navigation is done through TestProxy#getLocation. As you may see that it's straightforward for simple case and there is some custom navigation in case of Parameterized tests, because IDEA is able to run a test method with one parameter.

Hopefully it's clear how it works in IDEA. Please note, that our users work with different versions of junit (some of them still use junit3), so we have lots of reflection inside and would have to use it further.

If we have class name and method signature, then IDE is able to build the tests tree which corresponds to the "AST" structure of users code. If user or library expect another presentation, then this information should be somehow extractable from the description.

to api: I don't like the idea to navigate to {file:line} for java classes as I personally add new tests while view with already run tests is still open. This way I can see old failures and test further. That's why class_name::method_signature looks better for me.

out of scope of this discussion: we also use some private fields because we want sometimes to override default runners (e.g. if user select ignored test, he explicitly says that he wants to run it, so IDEA overrides "@Ignore" behaviour). Would be great to have api for that. I'll be happy to provide a digest of what we use via reflection.

Thanks! Anna

[kcooney]

@akozlova thanks for the detailed description. It'll take me a while to digest this, but so far I have twos questions:

1. does IDEA serialize Description objects?
2. would you mind creating a new issue in JUnit asking us to provide APIs for the cases where IDEA accesses private fields in JUnit? We had a few problems releasing 4.12 because IDEA was apparently accessing private fields via reflection, and we had changed the naming convention of private fields in 4.12.

Thanks!

[akozlova]

1. no, IDEA doesn't serialize Description objects: we pass method name and class name (because user's junit could differ from junit bundled in IDEA)
2. Sorry for that. I'll collect what we touch now but unfortunately there always would be old IDEAs which are not aware of the new api, after a couple of years probably...

Thanks

[jlink]

Many thanks, Anna, for giving us the details.

Are there any features that you'd like to see in IntelliJ or that are requested by IntelliJ users that would profit from a better and more powerful running API?

I am thinking along the lines of:

• Parallel test execution
• Continuous testing (like infinitest)
• Better test reporting
• Better test / assert / file navigation
• run parameterized tests with configurable parameters
• Freely composable test sessions (like ReSharper allows it)

I know that some of this is already possible via additional plug-ins, but there might be an opportunity for closer integration.

Johannes

2015-01-26 18:34 GMT+01:00 akozlova [email protected]:

1. no, IDEA doesn't serialize Description objects: we pass method name and class name (because user's junit could differ from junit bundled in IDEA)
2. Sorry for that. I'll collect what we touch now but unfortunately there always would be old IDEAs which are not aware of the new api, after a couple of years probably...

Thanks

— Reply to this email directly or view it on GitHub https://github.com/junit-team/junit/issues/1070#issuecomment-71501206.

[akozlova]

Johannes,

our users ask for parallel test executions, infinite execution of a test session and continuous testing indeed. And it would be great to have built-in support for it.

We have reporting based on XSLT transformations (I am not familiar with it but it works ;)). We have assert diff view (though it's based on pattern matching of strings, if it would be ComparisonExceptions it would be really helpful, e.g. junit.framework.Assert#failNotEquals). Parameterized tests and composable test sessions (you may choose tests to run based on patterns on fully qualified names and composition) are also already supported.

Anna

[kcooney]

If there are specific things we would like to do in JUnit (perhaps in the 5.0 timeframe) to help IDEA users, can we please file separate feature requests? I appreciate all of the information here, but I worry it will be lost in an issue that's focused on Description

[akozlova]

Created 1076/1077.

[Fuud]

Hi.

May be we can change type of Description.fUniqueId to String and add public getter => IDE will be able to filter tests by description. It will help to rerun failed tests in case of custom runners (for example Spock). But for better support we should change filtering logic in ParentRunner. It should also execute filter on children and not to filter-out parent if there is at least one non-filtered-out children.