INN
wp-cli may print errors/notices when json is requested, throwing python errors.
This is not necessarily a bug, but should be handled more gracefully.
Essentially, when wp tests are run, fab asks wp-cli for some json here and here.
If, for some reason, php puts up notices or errors they get printed before the json, example:
Notice: Constant WP_DEBUG already defined in phar:///usr/local/bin/wp/php/wp-cli.php(23) : eval()'d code on line 112
[{"name":"aspen"},{"name":"austinbulldog"}...
Same thing happens if a call to error_log() is made as the product of these calls. This becomes an issue if you want to var_log() something while writing a test.
When this happens, python can't decode the json and tests don't run. The error messages issued are super unhelpful.
I agree. In the event that json.loads can't parse the response from wp-cli, we should catch the exception (which should be a ValueError, I think) and print a friendly error message.
If you want to take this a step further, it might be possible to discern STDERROR from STDOUT when capturing output from the wp-cli commands. Then, if PHP issues a notice or warning (not a fatal error), we still have a chance at decoding the json returned by wp-cli and should be able to continue running tests.
