matomo-org
Steps to set up automated tests are incomplete
It's not currently possible to set up the following the instructions provided either here: https://developer.matomo.org/guides/getting-started-part-1 or here: https://developer.matomo.org/guides/tests-php
The steps/assumptions that are missing or incorrect are as follows:
- In the config, under [tests],
request_urimust be set up correctly or the tests won't run. - The default database name is
matomo_tests, notpiwik_testsas mentioned on the "Getting Started" page. - The docs should mention that the database user specified under [database_tests] should have CREATE/DROP DATABASE privileges - I think the underlying assumption is that everyone uses the root user.
- Just creating the tests database is not enough, it must be populated for the tests to run successfully - this can be achieved by running
./console tests:setup-fixture OmniFixture.
And once all of the above is done, it's time to run ./console tests:run unit, etc. I'd be very happy to create a pull request with the edited docs.
Additionally, there is code in https://github.com/matomo-org/matomo/blob/4.x-dev/tests/PHPUnit/bootstrap.php that should set up at least the config variables for the developer on the first run, however it doesn't actually work as it attempts to connect to the test database which is not yet set up - is it worth opening an issue for that or is it outdated code that should be scrapped rather than fixed?
Thanks @nina-py very appreciated. Already addresses two issues in https://github.com/matomo-org/developer-documentation/pull/386
The fourth point to run the omnifixture command should AFAIK not be needed generally.
With the request_uri setting I'm not so familiar. Which tests are you referring to here? Unit tests? Generally I see it's explained in https://developer.matomo.org/guides/tests-php and I wonder if it's best to have a complete guide in https://developer.matomo.org/guides/tests-php and only reference to this doc from the getting started guide instead of having an incomplete guide in the getting started section.
@tsteur, unit tests run fine without the request_uri specified, some integration and system tests run but many are skipped or are erroring out:
ninap@localhost matomo]$ ./console tests:run --testsuite integration
Executing command: cd /home/ninap/code/open-source/matomo/tests/PHPUnit && /home/ninap/code/open-source/matomo/vendor/bin/phpunit --testsuite IntegrationTests
PHPUnit 8.5.8 by Sebastian Bergmann and contributors.
Runtime: PHP 7.4.10
Configuration: /home/ninap/code/open-source/matomo/tests/PHPUnit/phpunit.xml.dist
ESSSSSESSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS 61 / 8368 ( 0%)
S.....ESSSSSSSEESSSSSSSSSSSESSSSSSSSSSSSSSESESSSSSSSSSSSSSSSE 122 / 8368 ( 1%)
[ninap@localhost matomo]$ ./console tests:run --testsuite system
[...]
EE...ESS...........................F.....F.FFFFEEESEFF.FF.FFF 61 / 1248 ( 4%)
I'll add a separate comment shortly re the Omnifixture thing.
Regarding the need to run the Omnifixture script: an empty test database is not enough because there is a bug in the bootstrap code, originating in the prepareServerVariables() function: https://github.com/matomo-org/matomo/blob/4.x-dev/tests/PHPUnit/bootstrap.php#L81
That function queries the main Matomo database to get an option:
$url = \Piwik\SettingsPiwik::getPiwikUrl();
but instead of a connection to the main Matomo database it tries to connect to the empty test database. The workaround is to run the Omnifixture script before running any tests.
PHP Fatal error: Uncaught PDOException: SQLSTATE[42S02]: Base table or view not found: 1146 Table 'dummy_db.option' doesn't exist in /home/ninap/code/open-source/matomo/libs/Zend/Db/Statement/Pdo.php:233
Stack trace:
#0 /home/ninap/code/open-source/matomo/libs/Zend/Db/Statement/Pdo.php(233): PDOStatement->execute()
#1 /home/ninap/code/open-source/matomo/libs/Zend/Db/Statement.php(300): Zend_Db_Statement_Pdo->_execute(Array)
#2 /home/ninap/code/open-source/matomo/libs/Zend/Db/Adapter/Abstract.php(479): Zend_Db_Statement->execute(Array)
#3 /home/ninap/code/open-source/matomo/libs/Zend/Db/Adapter/Pdo/Abstract.php(238): Zend_Db_Adapter_Abstract->query('SELECT option_v...', Array)
#4 /home/ninap/code/open-source/matomo/core/Db/Adapter/Pdo/Mysql.php(309): Zend_Db_Adapter_Pdo_Abstract->query('SELECT option_v...', Array)
#5 /home/ninap/code/open-source/matomo/libs/Zend/Db/Adapter/Abstract.php(736): Piwik\Db\Adapter\Pdo\Mysql->query('SELECT option_v...', Array)
#6 /home/ninap/code/open-source/matomo/core/Db.php(297): Zend_Db_Adapter_A in /home/ninap/code/open-source/matomo/libs/Zend/Db/Statement/Pdo.php on line 236
Fatal error: Uncaught PDOException: SQLSTATE[42S02]: Base table or view not found: 1146 Table 'dummy_db.option' doesn't exist in /home/ninap/code/open-source/matomo/libs/Zend/Db/Statement/Pdo.php on line 236
Zend_Db_Statement_Exception: SQLSTATE[42S02]: Base table or view not found: 1146 Table 'dummy_db.option' doesn't exist in /home/ninap/code/open-source/matomo/libs/Zend/Db/Statement/Pdo.php on line 236
Call Stack:
0.0004 363136 1. {main}() /home/ninap/code/open-source/matomo/vendor/phpunit/phpunit/phpunit:0
0.0181 444728 2. PHPUnit\TextUI\Command::main() /home/ninap/code/open-source/matomo/vendor/phpunit/phpunit/phpunit:61
0.0181 444856 3. PHPUnit\TextUI\Command->run() /home/ninap/code/open-source/matomo/vendor/phpunit/phpunit/src/TextUI/Command.php:163
0.0181 444856 4. PHPUnit\TextUI\Command->handleArguments() /home/ninap/code/open-source/matomo/vendor/phpunit/phpunit/src/TextUI/Command.php:171
0.0257 454128 5. PHPUnit\TextUI\Command->handleBootstrap() /home/ninap/code/open-source/matomo/vendor/phpunit/phpunit/src/TextUI/Command.php:888
0.0259 454240 6. PHPUnit\Util\FileLoader::checkAndLoad() /home/ninap/code/open-source/matomo/vendor/phpunit/phpunit/src/TextUI/Command.php:1091
0.0260 454496 7. PHPUnit\Util\FileLoader::load() /home/ninap/code/open-source/matomo/vendor/phpunit/phpunit/src/Util/FileLoader.php:47
0.0263 455448 8. include_once('/home/ninap/code/open-source/matomo/tests/PHPUnit/bootstrap.php') /home/ninap/code/open-source/matomo/vendor/phpunit/phpunit/src/Util/FileLoader.php:59
0.1244 1550712 9. prepareServerVariables() /home/ninap/code/open-source/matomo/tests/PHPUnit/bootstrap.php:135
0.1244 1550744 10. Piwik\SettingsPiwik::getPiwikUrl() /home/ninap/code/open-source/matomo/tests/PHPUnit/bootstrap.php:87
0.1244 1550744 11. Piwik\Option::get() /home/ninap/code/open-source/matomo/core/SettingsPiwik.php:181
0.1244 1550744 12. Piwik\Option->getValue() /home/ninap/code/open-source/matomo/core/Option.php:48
0.1244 1550744 13. Piwik\Option->autoload() /home/ninap/code/open-source/matomo/core/Option.php:178
0.1244 1550840 14. Piwik\Db::fetchAll() /home/ninap/code/open-source/matomo/core/Option.php:288
Thanks @nina-py . I guess the idea was to configure the request_uri automatically but it might not work because of the bootstrap bug. I think in this case it should have actually connected to the actual Matomo (not the test DB). And if Matomo is installed it will be automatically configured. Is it possible that Matomo wasn't installed separately? In this case we could maybe catch an exception and mention to configure request_uri setting?
@tsteur, yes, I think that's what the bootstrap code is meant to do - to copy the DB credentials from the main DB and autoconfigure the request_uri parameter. I have a separate installation of Matomo and I made sure I was able to reproduce this on a brand new installation in case there was something dodgy in my local setup (perhaps there still is! :)).
There is actually code at the bottom of the bootstrap file which tells you off if you haven't configured the request_uri setting, so that is taken care of already: https://github.com/matomo-org/matomo/blob/e0019128ed33d321185a5ea283090f7541c2b83d/tests/PHPUnit/bootstrap.php#L139 . It's just the database part that I'm confused with - if you set up a brand new Matomo instance, can you replicate the issues with the database I've ran into?
@nina-py I can definitely reproduce it and I can't even think of a fix.
I'm thinking the only way we can fix this is to throw a useful exception if request_uri is not configured and instructing people to configure it.