neowit
running tests synchronously doesn't terminate
This is a continuation of the issue from https://github.com/neowit/tooling-force.com/issues/20#issuecomment-269706041
Looks like it might not be an issue directly with tooling-force.com but rather the salesforce tooling api.
I've been reading some of the tooling api docs and it looks like there were changes at v37.0
I noticed that this project uses 36.0 https://github.com/neowit/tooling-force.com/blob/master/src/main/scala/com/neowit/utils/Config.scala#L206
I've been reading some of the tooling api docs and it looks like there were changes at v37.0
What changes are you referring to please? Is there something specific to synchronous test run ?
I noticed that this project uses 36.0
I have no plans to update to v38 just yet because of this and this.
Oh, you know what I didn't realize this tool used the soap api but the docs I was reading were for the rest api. https://developer.salesforce.com/docs/atlas.en-us.api_tooling.meta/api_tooling/intro_rest_resources.htm has changes for the test running endpoint for v37.0
I tried another test this time running without quotes and I got this error. It might be related to me cancelling the previous sync test run but it is still running on the salesforce side. I also removed the quotes around the * this time.
$ java -jar tooling-force.com-0.3.6.7-20.jar --action=runTestsTooling --config=$HOME/.apex/login/sandbox-tomtom.properties --projectPath=$HOME/work/salesforce --async=false --testsToRun=* --responseFilePath=/dev/null --pollWaitMillis=1000 --maxPollRequests=1000
[INFO] RunTests - Run tests Synchronous ...
[INFO] RunTests - Run tests Synchronous ... # elapsed: 3s
[INFO] RunTests - Run tests Synchronous ... # elapsed: 6s
[INFO] RunTests - Run tests Synchronous ... # elapsed: 9s
[INFO] RunTests - Run tests Synchronous ... # elapsed: 12s
[INFO] RunTests - Run tests Synchronous ... # elapsed: 15s
[INFO] RunTests - Run tests Synchronous ... # elapsed: 18s
[INFO] RunTests - Run tests Synchronous ... # elapsed: 21s
[INFO] RunTests - Run tests Synchronous ... # elapsed: 24s
[INFO] RunTests - Run tests Synchronous ... # elapsed: 27s
[INFO] RunTests - Run tests Synchronous ... # elapsed: 30s
[ERROR] ResponseWriter - [UnexpectedErrorFault [ApiFault exceptionCode='UNKNOWN_EXCEPTION'
[ERROR] ResponseWriter - exceptionMessage='admin operation already in progress'
[ERROR] ResponseWriter - upgradeURL='null'
[ERROR] ResponseWriter - upgradeMessage='null'
[ERROR] ResponseWriter - ]
[ERROR] ResponseWriter - ]
[ERROR] ResponseWriter -
[ERROR] ResponseWriter - at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
[ERROR] ResponseWriter - at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
[ERROR] ResponseWriter - at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
[ERROR] ResponseWriter - at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
[ERROR] ResponseWriter - at java.lang.Class.newInstance(Class.java:433)
[ERROR] ResponseWriter - at com.sforce.ws.bind.TypeMapper.readSingle(TypeMapper.java:673)
[ERROR] ResponseWriter - at com.sforce.ws.bind.TypeMapper.readObject(TypeMapper.java:548)
[ERROR] ResponseWriter - at com.sforce.ws.transport.SoapConnection.parseDetail(SoapConnection.java:236)
[ERROR] ResponseWriter - at com.sforce.ws.transport.SoapConnection.createException(SoapConnection.java:210)
[ERROR] ResponseWriter - at com.sforce.ws.transport.SoapConnection.receive(SoapConnection.java:156)
[ERROR] ResponseWriter - at com.sforce.ws.transport.SoapConnection.send(SoapConnection.java:99)
[ERROR] ResponseWriter - at com.sforce.soap.tooling.ToolingConnection.runTests(ToolingConnection.java:502)
[ERROR] ResponseWriter - at com.neowit.apex.Session$$anonfun$38.apply(Session.scala:1071)
[ERROR] ResponseWriter - at com.neowit.apex.Session$$anonfun$38.apply(Session.scala:1069)
[ERROR] ResponseWriter - at com.neowit.apex.Session.withRetry(Session.scala:722)
[ERROR] ResponseWriter - at com.neowit.apex.Session.runTestsTooling(Session.scala:1069)
[ERROR] ResponseWriter - at com.neowit.apex.actions.tooling.RunTests$$anonfun$runTestsSynchronous$1.apply$mcV$sp(RunTests.scala:230)
[ERROR] ResponseWriter - at com.neowit.apex.actions.tooling.RunTests$$anonfun$runTestsSynchronous$1.apply(RunTests.scala:230)
[ERROR] ResponseWriter - at com.neowit.apex.actions.tooling.RunTests$$anonfun$runTestsSynchronous$1.apply(RunTests.scala:230)
[ERROR] ResponseWriter - at com.neowit.utils.Logging$.repeatingInfo(Logging.scala:101)
[ERROR] ResponseWriter - at com.neowit.apex.actions.tooling.RunTests.runTestsSynchronous(RunTests.scala:231)
[ERROR] ResponseWriter - at com.neowit.apex.actions.tooling.RunTests.runTests(RunTests.scala:212)
[ERROR] ResponseWriter - at com.neowit.apex.actions.tooling.RunTests.act(RunTests.scala:151)
[ERROR] ResponseWriter - at com.neowit.apex.actions.Action$class.execute(Action.scala:91)
[ERROR] ResponseWriter - at com.neowit.apex.actions.ApexActionWithWritableSession.execute(Action.scala:258)
[ERROR] ResponseWriter - at com.neowit.apex.Executor.run(Runner.scala:136)
[ERROR] ResponseWriter - at com.neowit.apex.Executor.execute(Runner.scala:49)
[ERROR] ResponseWriter - at com.neowit.apex.Runner$.main(Runner.scala:33)
[ERROR] ResponseWriter - at com.neowit.apex.Runner.main(Runner.scala)
[INFO] Executor - # Time taken: 31.229s
It might be related to me cancelling the previous sync test run but it is still running on the salesforce side.
According to the error message you are most likely right.
When you kill tooling-force.com.jar it does not stop processes already running on SFDC side. Cancelling long running processes even via SFDC UI does not always work so implementing some sort of graceful shutdown on tooling-force.com.jar side would probably be a waste of effort.
I left it running over night and it got to 32314s before i cancelled the job.
I have done few test of my own. It looks like SFDC does not like running multiple test classes in synchronous mode with tooling api. On the other hand it seems to have no problem running multiple/all tests via Metadata API or tooling-async. In my tests I was able to run multiple test classes with tooling-sync (in a reasonable amount of time) only when my tests were very basic and did not touch DB. As soon as tests become more realistic (and include DB changes) their execution time (with tooling-sync) increased exponentially, but they did finish eventually.
Not sure what to do with this. Simply disallowing multiple tests with tooling-sync could be an option, but there is always a chance that new SFDC API handles those better.
Out of interest - is there a specific reason why you have to run tests with tooling-sync, or you just want to help me debug this (while you actual workflow can be covered by running tests with metadata API or tooling-async)?
is there a specific reason why you have to run tests with tooling-sync, or you just want to help me debug this
at first i wanted to help debug the tool, unfortunately it looks like it is a salesforce tooling api issue the only thing i can think of to try right now is to use the latest api version and see if that helps
metadata API
Yes this looks like the only option available that currently works. I was avoiding this because you need the actual project to do a test deploy while running all tests.
tooling-async
I want to avoid using async testing because salesforce will fail tests in this mode that it wouldn't fail in sync mode. e.g. the infamous unable to lock row error. I'm trying to do regression testing so I want to avoid getting alerted on false positives.
Thank you for the clarification. I will continue researching this when I get a chance.
I want to avoid getting alerted on false positives.
FYI - "Unable to lock row" reported during tests is usually an indication that same problem may happen in production. One exception I am aware of (when this issue in Test does not mean issue in Prod ) is when tests have to setup custom settings. In production people usually do not run concurrent updates to custom setting objects.
Just a random thought - do you know what happens if you use tooling-async, but prior to executing tests Disable Parallel Apex Testing in Your Name > Setup > Develop > Apex Test Execution > Options:
Does salesforce respect this ?
FYI - here is what documentation has about runTestsSynchronous https://developer.salesforce.com/docs/atlas.en-us.api_tooling.meta/api_tooling/intro_rest_resources.htm?search_text=runTestsSynchronous
All test methods in a synchronous test run must be in the same class.
I did not notice this note before. Probably was there all the time. The fact that it is sometimes possible to run tests from multiple classes in synchronous mode is probably considered a bug in SFDC test engine, if we take documentation into account.
good catch.. looks like there isn't really anything more we really have to do for this since i was under the false assumption that you could use the tooling api to run all tests synchronously. The workaround you mentioned is the only way I can think of without creating a fake deploy method.
FYI - recent discussion from Stackexchange which may be of interest - regarding a way to "Disable Parallel Apex Testing" programmatically, to ensure it is in the correct state before you run tests.
http://salesforce.stackexchange.com/questions/155502/tooling-api-enable-disable-parallel-test-execution
It confirms that there is no API for this, bit if one is adventurous they could try HTTP POST method described in the accepted answer.