StackStorm
Error in st2-run-pack-tests when running with st2 3.9dev images
SUMMARY
Testing st2-run-pack-tests with st2 3.9dev seems to be broken
STACKSTORM VERSION
Paste the output of st2 --version:
/opt/stackstorm# st2 --version
st2 3.9dev (49f61fc), on Python 3.10.12
OS, environment, install method
Docker method of installation with st2 3.9dev tagged images
Steps to reproduce the problem
I have used following command (which i have been using in st2 3.8 so far) to run the test case e.g. for a pack named PACK_NAME, the error seems to be related to pytest -
(PACK_NAME) /opt/stackstorm# st2-run-pack-tests -p packs/PACK_NAME/ -c -f TEST CASE
- First attempt : Tried running above command which includes "-c" for coverage - Fails with error related to
ERROR: usage: pytest [options] [file_or_dir] [file_or_dir] [...]
pytest: error: unrecognized arguments: --cov=/opt/stackstorm/packs/PACK_NAME/img --cov=/opt/stackstorm/packs/PACK_NAME/.git --cov=/opt/stackstorm/packs/PACK_NAME/docs --cov=/opt/stackstorm/packs/PACK_NAME/flowcharts --cov=/opt/stackstorm/packs/PACK_NAME/rules --cov=/opt/stackstorm/packs/PACK_NAME/actions --cov=/opt/stackstorm/packs/PACK_NAME/lib --cov=/opt/stackstorm/packs/PACK_NAME/config
inifile: None
rootdir: /opt/stackstorm/packs/PACK_NAME
- Second attempt: Dropped "-c" to check if it runs without it - Fails with error that test case file is not located (which is located at the path of the pack)
Running tests...
=================================================================================== test session starts ===================================================================================
platform linux -- Python 3.10.12, pytest-7.0.1, pluggy-1.5.0 -- /opt/stackstorm/st2/bin/python
cachedir: .pytest_cache
rootdir: /opt/stackstorm/packs/PACK_NAME
collected 0 items
================================================================================== no tests ran in 0.00s ==================================================================================
ERROR: file or directory not found: Test_file:TestCaseClass:testfunction
Expected Results
I expect test cases to run normally as mentioned in https://docs.stackstorm.com/development/pack_testing.html?#running-tests
Actual Results
Attempt 1 -
(PACK_NAME) /opt/stackstorm# st2-run-pack-tests -p packs/PACK_NAME/ -c -f TESTCASE
Running tests for pack: PACK_NAME
Creating virtualenv in /tmp/st2-pack-tests-virtualenvs/PACK_NAME...
created virtual environment CPython3.10.12.final.0-64 in 273ms
creator CPython3Posix(dest=/tmp/st2-pack-tests-virtualenvs/PACK_NAME, clear=False, no_vcs_ignore=False, global=True)
seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/root/.local/share/virtualenv)
added seed packages: pip==25.1.1, setuptools==80.9.0, wheel==0.45.1
activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
Activating virtualenv in /tmp/st2-pack-tests-virtualenvs/PACK_NAME...
Collecting pip==20.3.3
Downloading pip-20.3.3-py2.py3-none-any.whl.metadata (4.3 kB)
Downloading pip-20.3.3-py2.py3-none-any.whl (1.5 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.5/1.5 MB 26.7 MB/s eta 0:00:00
Installing collected packages: pip
Attempting uninstall: pip
Found existing installation: pip 25.1.1
Uninstalling pip-25.1.1:
Successfully uninstalled pip-25.1.1
Successfully installed pip-20.3.3
Installing global pack test dependencies...
WARNING: You are using pip version 20.3.3; however, version 25.1.1 is available.
You should consider upgrading via the '/tmp/st2-pack-tests-virtualenvs/PACK_NAME/bin/python -m pip install --upgrade pip' command.
Installing pack-specific dependencies...
WARNING: You are using pip version 20.3.3; however, version 25.1.1 is available.
You should consider upgrading via the '/tmp/st2-pack-tests-virtualenvs/PACK_NAME/bin/python -m pip install --upgrade pip' command.
Running tests...
/opt/stackstorm/st2/lib/python3.10/site-packages/_pytest/config/__init__.py:325: PluggyTeardownRaisedWarning: A plugin raised an exception during an old-style hookwrapper teardown.
Plugin: helpconfig, Hook: pytest_cmdline_parse
UsageError: usage: pytest [options] [file_or_dir] [file_or_dir] [...]
pytest: error: unrecognized arguments: --cov=/opt/stackstorm/packs/PACK_NAME/img --cov=/opt/stackstorm/packs/PACK_NAME/.git --cov=/opt/stackstorm/packs/PACK_NAME/docs --cov=/opt/stackstorm/packs/PACK_NAME/flowcharts --cov=/opt/stackstorm/packs/PACK_NAME/rules --cov=/opt/stackstorm/packs/PACK_NAME/actions --cov=/opt/stackstorm/packs/PACK_NAME/lib --cov=/opt/stackstorm/packs/PACK_NAME/config
inifile: None
rootdir: /opt/stackstorm/packs/PACK_NAME
For more information see https://pluggy.readthedocs.io/en/stable/api_reference.html#pluggy.PluggyTeardownRaisedWarning
config = pluginmanager.hook.pytest_cmdline_parse(
ERROR: usage: pytest [options] [file_or_dir] [file_or_dir] [...]
pytest: error: unrecognized arguments: --cov=/opt/stackstorm/packs/PACK_NAME/img --cov=/opt/stackstorm/packs/PACK_NAME/.git --cov=/opt/stackstorm/packs/PACK_NAME/docs --cov=/opt/stackstorm/packs/PACK_NAME/flowcharts --cov=/opt/stackstorm/packs/PACK_NAME/rules --cov=/opt/stackstorm/packs/PACK_NAME/actions --cov=/opt/stackstorm/packs/PACK_NAME/lib --cov=/opt/stackstorm/packs/PACK_NAME/config
inifile: None
rootdir: /opt/stackstorm/packs/PACK_NAME
Attempt 2 -
/opt/stackstorm# st2-run-pack-tests -p /opt/stackstorm/packs/PACK_NAME/ -f Test_file:TestCaseClass:testfunction
Running tests for pack: PACK_NAME
Creating virtualenv in /tmp/st2-pack-tests-virtualenvs/PACK_NAME...
created virtual environment CPython3.10.12.final.0-64 in 356ms
creator CPython3Posix(dest=/tmp/st2-pack-tests-virtualenvs/PACK_NAME, clear=False, no_vcs_ignore=False, global=True)
seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/root/.local/share/virtualenv)
added seed packages: ...
activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
Activating virtualenv in /tmp/st2-pack-tests-virtualenvs/PACK_NAME...
Collecting pip==20.3.3
Using cached pip-20.3.3-py2.py3-none-any.whl.metadata (4.3 kB)
Using cached pip-20.3.3-py2.py3-none-any.whl (1.5 MB)
Installing collected packages: pip
Attempting uninstall: pip
Found existing installation: pip 25.1.1
Uninstalling pip-25.1.1:
Successfully uninstalled pip-25.1.1
Successfully installed pip-20.3.3
Installing global pack test dependencies...
Installing pack-specific dependencies...
WARNING: You are using pip version 20.3.3; however, version 25.1.1 is available.
You should consider upgrading via the '/tmp/st2-pack-tests-virtualenvs/PACK_NAME/bin/python -m pip install --upgrade pip' command.
Running tests...
=================================================================================== test session starts ===================================================================================
platform linux -- Python 3.10.12, pytest-7.0.1, pluggy-1.5.0 -- /opt/stackstorm/st2/bin/python
cachedir: .pytest_cache
rootdir: /opt/stackstorm/packs/PACK_NAME
collected 0 items
================================================================================== no tests ran in 0.00s ==================================================================================
ERROR: file or directory not found: Test_file:TestCaseClass:testfunction
Thanks!
@NitinNahal Any idea of a fix for this?
Currently i am not sure of what causes such an error. Can it be related to internal migration to pytest ?
There's a good chance it's related, but all the pytest work is was completed with tests passing. There might have been something overlooked in st2-run-pack-tests that needs fixing for pytest to work correctly.
Hmmm, trying to gather the changes done to this script recently, i could just see the change was part of -
https://github.com/StackStorm/st2/pull/6291
https://github.com/StackStorm/st2/pull/6291/commits/d356a751727672f215f2e58adf6a52a456368feb
That looks like that switch to pytest. Any idea what other changes might be required to get the script working again?
Hello @nzlosh , Yes spent some time today to check, there seems to be multiple small issues with the script which seems to have been missed during migration. Also there might be a requirement to update the docs on how we run this script when a file has to be specified.
I will try to create a PR for the changes I found in this script, since now I am able to run the test as expected. Also can you let me know how can we update something on docs for this section if I want - https://docs.stackstorm.com/development/pack_testing.html?#running-tests
@NitinNahal that's great news! Thank you for looking into that. You can update the documentation by opening a PR against the documentation repository https://github.com/StackStorm/st2docs
Following two PRs for the same -
st2 : https://github.com/StackStorm/st2/pull/6340 st2docs: https://github.com/StackStorm/st2docs/pull/1200
I hope we can include them for 3.9 release.
Hey @nzlosh can you help to review these PRs ? or if you could help to get it reviewed by some other TSC members Also do we have any tentative timeline for when 3.9 release will be made officially ?