pytest-dev
`'store_true'` does not work correctly in parser.addoption
One more from https://github.com/conda-forge/staged-recipes/pull/16888, where (after patching in workarounds for #9300 & #9301), the test suite failed with
==================================== ERRORS ====================================
________________ ERROR at setup of test_serialization[cpu-gelu] ________________
self = <_pytest.config.Config object at 0x7fc1975d41c0>
name = 'skip_custom_ops', default = <NOTSET>, skip = False
def getoption(self, name: str, default=notset, skip: bool = False):
"""Return command line option value.
:param name: Name of the option. You may also specify
the literal ``--OPT`` option instead of the "dest" option name.
:param default: Default value if no option of that name exists.
:param skip: If True, raise pytest.skip if option does not exists
or has a None value.
"""
name = self._opt2dest.get(name, name)
try:
> val = getattr(self.option, name)
E AttributeError: 'Namespace' object has no attribute 'skip_custom_ops'
../[...]/_pytest/config/__init__.py:1463: AttributeError
The above exception was the direct cause of the following exception:
request = <SubRequest 'set_global_variables' for <Function test_serialization[cpu-gelu]>>
@pytest.fixture(scope="session", autouse=True)
def set_global_variables(request):
> if request.config.getoption("--skip-custom-ops"):
E ValueError: no option named 'skip_custom_ops
So from that, it looks like something is going wrong with skip_custom_ops. Chasing around the upstream repo I find that this option is added here as follows:
def pytest_addoption(parser):
parser.addoption(
"--skip-custom-ops",
action="store_true",
help="When a custom op is being loaded in a test, skip this test.",
)
The pytest docs for parser.addoption say that this works like argparse, where the linked docs say:
'store_true'and'store_false'- These are special cases of'store_const'used for storing the valuesTrueandFalserespectively. In addition, they create default values ofFalseandTruerespectively.
What caught my eye in the stack trace was the default = <NOTSET>, which is clearly at odds with that documentation. To verify this hypothesis, I added the following patch
diff --git a/tensorflow_addons/utils/test_utils.py b/tensorflow_addons/utils/test_utils.py
index 9d1dec0..0efd707 100644
--- a/tensorflow_addons/utils/test_utils.py
+++ b/tensorflow_addons/utils/test_utils.py
@@ -134,7 +134,8 @@ def set_seeds():
def pytest_addoption(parser):
parser.addoption(
"--skip-custom-ops",
- action="store_true",
+ action="store",
+ default=True,
help="When a custom op is being loaded in a test, skip this test.",
)
and voilà, it works.
@h-vetinari I've tried to reproduce the error, by writing a test. When this failed I've cloned the repo and wrote a simple test just to debug the parser, yet not error was presented, may you please add more info (python version, package version) thanks!
I think running the upstream repo is the wrong approach. The issue is about store_true not being respected, which is independent of a given package setup. The tensorflow-stack does a lot of unusual things, and may not be exposed due to various quirks around the build- & test-setup, but I'm definitely hitting this in https://github.com/conda-forge/staged-recipes/pull/16888. Still I reran this again by commenting out the patch from the OP, and indeed it keeps failing (here's a link to the respective CI run; look under "linux_64_centos7").
The versions in question are:
pytest: 6.2.5-py39hf3d152e_1 conda-forge
python: 3.9.7-hb7a2778_3_cpython conda-forge
tensorflow-addons: 0.14.0-py39hfe13ede_0 local
@h-vetinari So I've tried to mimic the problem above, I wrote a test:
def test_config_getoption_store(self, pytester: Pytester) -> None:
pytester.makeconftest(
"""
def pytest_addoption(parser):
parser.addoption(
"--skip-custom-ops",
action="store_true",
help="When a custom op is being loaded in a test, skip this test.",
)
"""
)
config = pytester.parseconfig("-v")
assert not config.getoption("--skip-custom-ops")
yet no luck in reproducing the issue, the test seems to pass:
============================================================================== test session starts ==============================================================================
platform linux -- Python 3.9.7, pytest-6.2.5, py-1.11.0, pluggy-1.0.0 -- /.venv/bin/python3.9
cachedir: .pytest_cache
rootdir: pytest
plugins: typeguard-2.13.2
collected 1 item
testing/test_config.py::TestConfigAPI::test_config_getoption_store PASSED [100%]
=============================================================================== 1 passed in 0.55s ===============================================================================
Do you have any idea of other quirks that might affect the environment?
Hey @yuvalshi0
Thanks for digging into this - no idea where this discrepancy is coming from, but you could try to download the artefact from here (assuming you're on linux, as the test session above seems to indicate), unzip it, install it as follows
conda create -n test_env -c "path/to/where/you/unpacked/the/artefact" -c conda-forge tensorflow-addons python=3.8
and then try to cd into a checkout of tensorflow-addons and run the tests.
Still no luck unfortunately, under this conda environment the tests till passes:
# packages in environment at /root/anaconda3/envs/test_env:
#
# Name Version Build Channel
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 1_gnu conda-forge
abseil-cpp 20210324.2 h9c3ff4c_0 conda-forge
absl-py 0.15.0 pyhd8ed1ab_0 conda-forge
aiohttp 3.8.1 py39h3811e60_0 conda-forge
aiosignal 1.2.0 pyhd8ed1ab_0 conda-forge
appdirs 1.4.4 pyh9f0ad1d_0 conda-forge
astunparse 1.6.3 pyhd8ed1ab_0 conda-forge
async-timeout 4.0.1 pyhd8ed1ab_0 conda-forge
attrs 21.2.0 pyhd8ed1ab_0 conda-forge
blinker 1.4 py_1 conda-forge
blosc 1.21.0 h9c3ff4c_0 conda-forge
brotli 1.0.9 h7f98852_6 conda-forge
brotli-bin 1.0.9 h7f98852_6 conda-forge
brotlipy 0.7.0 py39h3811e60_1003 conda-forge
brunsli 0.1 h9c3ff4c_0 conda-forge
bzip2 1.0.8 h7f98852_4 conda-forge
c-ares 1.18.1 h7f98852_0 conda-forge
c-blosc2 2.0.4 h5f21a17_1 conda-forge
ca-certificates 2021.10.8 ha878542_0 conda-forge
cached-property 1.5.2 hd8ed1ab_1 conda-forge
cached_property 1.5.2 pyha770c72_1 conda-forge
cachetools 4.2.4 pyhd8ed1ab_0 conda-forge
certifi 2021.10.8 py39hf3d152e_1 conda-forge
cffi 1.15.0 py39h4bc2ebd_0 conda-forge
cfitsio 4.0.0 h9a35b8e_0 conda-forge
charls 2.2.0 h9c3ff4c_0 conda-forge
charset-normalizer 2.0.8 pyhd8ed1ab_0 conda-forge
click 8.0.3 py39hf3d152e_1 conda-forge
cloudpickle 2.0.0 pyhd8ed1ab_0 conda-forge
colorama 0.4.4 pyh9f0ad1d_0 conda-forge
cryptography 35.0.0 py39h95dcef6_2 conda-forge
cycler 0.11.0 pyhd8ed1ab_0 conda-forge
cytoolz 0.11.2 py39h3811e60_1 conda-forge
dask-core 2021.11.2 pyhd8ed1ab_0 conda-forge
dataclasses 0.8 pyhc8e2a94_3 conda-forge
fonttools 4.28.2 py39h3811e60_0 conda-forge
freetype 2.10.4 h0708190_1 conda-forge
frozenlist 1.2.0 py39h3811e60_1 conda-forge
fsspec 2021.11.1 pyhd8ed1ab_0 conda-forge
gast 0.4.0 pyh9f0ad1d_0 conda-forge
giflib 5.2.1 h36c2ea0_2 conda-forge
google-auth 1.35.0 pyh6c4a22f_0 conda-forge
google-auth-oauthlib 0.4.6 pyhd8ed1ab_0 conda-forge
google-pasta 0.2.0 pyh8c360ce_0 conda-forge
grpc-cpp 1.40.0 h05f19cf_2 conda-forge
grpcio 1.40.0 py39hff7568b_0 conda-forge
h5py 3.1.0 nompi_py39h25020de_100 conda-forge
hdf5 1.10.6 nompi_h6a2412b_1114 conda-forge
icu 69.1 h9c3ff4c_0 conda-forge
idna 3.1 pyhd3deb0d_0 conda-forge
imagecodecs 2021.11.20 py39h571908b_1 conda-forge
imageio 2.9.0 py_0 conda-forge
importlib-metadata 4.8.2 py39hf3d152e_0 conda-forge
iniconfig 1.1.1 pyh9f0ad1d_0 conda-forge
jbig 2.1 h7f98852_2003 conda-forge
joblib 1.1.0 pyhd8ed1ab_0 conda-forge
jpeg 9d h36c2ea0_0 conda-forge
jxrlib 1.1 h7f98852_2 conda-forge
keras 2.6.0 pyhd8ed1ab_1 conda-forge
keras-preprocessing 1.1.2 pyhd8ed1ab_0 conda-forge
kiwisolver 1.3.2 py39h1a9c180_1 conda-forge
krb5 1.19.2 hcc1bbae_3 conda-forge
lcms2 2.12 hddcbb42_0 conda-forge
ld_impl_linux-64 2.36.1 hea4e1c9_2 conda-forge
lerc 3.0 h9c3ff4c_0 conda-forge
libaec 1.0.6 h9c3ff4c_0 conda-forge
libblas 3.9.0 12_linux64_openblas conda-forge
libbrotlicommon 1.0.9 h7f98852_6 conda-forge
libbrotlidec 1.0.9 h7f98852_6 conda-forge
libbrotlienc 1.0.9 h7f98852_6 conda-forge
libcblas 3.9.0 12_linux64_openblas conda-forge
libcurl 7.80.0 h2574ce0_0 conda-forge
libdeflate 1.8 h7f98852_0 conda-forge
libedit 3.1.20191231 he28a2e2_2 conda-forge
libev 4.33 h516909a_1 conda-forge
libffi 3.4.2 h7f98852_5 conda-forge
libgcc-ng 11.2.0 h1d223b6_11 conda-forge
libgfortran-ng 11.2.0 h69a702a_11 conda-forge
libgfortran5 11.2.0 h5c6108e_11 conda-forge
libgomp 11.2.0 h1d223b6_11 conda-forge
liblapack 3.9.0 12_linux64_openblas conda-forge
libnghttp2 1.43.0 h812cca2_1 conda-forge
libopenblas 0.3.18 pthreads_h8fe5266_0 conda-forge
libpng 1.6.37 h21135ba_2 conda-forge
libprotobuf 3.18.1 h780b84a_0 conda-forge
libssh2 1.10.0 ha56f1ee_2 conda-forge
libstdcxx-ng 11.2.0 he4da1e4_11 conda-forge
libtensorflow_cc 2.6.2 cpu_hfd9b42a_0 conda-forge
libtiff 4.3.0 h6f004c6_2 conda-forge
libwebp-base 1.2.1 h7f98852_0 conda-forge
libzlib 1.2.11 h36c2ea0_1013 conda-forge
libzopfli 1.0.3 h9c3ff4c_0 conda-forge
locket 0.2.0 py_2 conda-forge
lz4-c 1.9.3 h9c3ff4c_1 conda-forge
markdown 3.3.6 pyhd8ed1ab_0 conda-forge
matplotlib-base 3.5.0 py39h2fa2bec_0 conda-forge
more-itertools 8.12.0 pyhd8ed1ab_0 conda-forge
multidict 5.2.0 py39h3811e60_1 conda-forge
munkres 1.1.4 pyh9f0ad1d_0 conda-forge
ncurses 6.2 h58526e2_4 conda-forge
networkx 2.6.3 pyhd8ed1ab_1 conda-forge
numpy 1.19.5 py39hdbf815f_2 conda-forge
oauthlib 3.1.1 pyhd8ed1ab_0 conda-forge
olefile 0.46 pyh9f0ad1d_1 conda-forge
openjpeg 2.4.0 hb52868f_1 conda-forge
openssl 1.1.1l h7f98852_0 conda-forge
opt_einsum 3.3.0 pyhd8ed1ab_1 conda-forge
packaging 21.3 pyhd8ed1ab_0 conda-forge
pandas 1.3.4 py39hde0f152_1 conda-forge
partd 1.2.0 pyhd8ed1ab_0 conda-forge
pillow 8.4.0 py39ha612740_0 conda-forge
pip 21.3.1 pyhd8ed1ab_0 conda-forge
pluggy 1.0.0 py39hf3d152e_2 conda-forge
pooch 1.5.2 pyhd8ed1ab_0 conda-forge
protobuf 3.18.1 py39he80948d_0 conda-forge
py 1.11.0 pyh6c4a22f_0 conda-forge
pyasn1 0.4.8 py_0 conda-forge
pyasn1-modules 0.2.7 py_0 conda-forge
pycparser 2.21 pyhd8ed1ab_0 conda-forge
pyjwt 2.3.0 pyhd8ed1ab_0 conda-forge
pyopenssl 21.0.0 pyhd8ed1ab_0 conda-forge
pyparsing 3.0.6 pyhd8ed1ab_0 conda-forge
pysocks 1.7.1 py39hf3d152e_4 conda-forge
pytest 6.2.5 py39hf3d152e_1 conda-forge
python 3.9.7 hb7a2778_3_cpython conda-forge
python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge
python-flatbuffers 1.12 pyhd8ed1ab_1 conda-forge
python_abi 3.9 2_cp39 conda-forge
pytz 2021.3 pyhd8ed1ab_0 conda-forge
pyu2f 0.1.5 pyhd8ed1ab_0 conda-forge
pywavelets 1.2.0 py39hce5d2b2_0 conda-forge
pyyaml 6.0 py39h3811e60_3 conda-forge
re2 2021.09.01 h9c3ff4c_0 conda-forge
readline 8.1 h46c0cb4_0 conda-forge
requests 2.26.0 pyhd8ed1ab_1 conda-forge
requests-oauthlib 1.3.0 pyh9f0ad1d_0 conda-forge
rsa 4.8 pyhd8ed1ab_0 conda-forge
scikit-image 0.18.3 py39hde0f152_0 conda-forge
scikit-learn 1.0.1 py39h4dfa638_2 conda-forge
scipy 1.7.3 py39hee8e79c_0 conda-forge
setuptools 59.2.0 py39hf3d152e_0 conda-forge
six 1.15.0 pyh9f0ad1d_0 conda-forge
snappy 1.1.8 he1b5a44_3 conda-forge
sqlite 3.37.0 h9cd32fc_0 conda-forge
tensorboard 2.6.0 pyhd8ed1ab_1 conda-forge
tensorboard-data-server 0.6.0 py39h95dcef6_1 conda-forge
tensorboard-plugin-wit 1.8.0 pyh44b312d_0 conda-forge
tensorflow 2.6.2 cpu_py39h1b7c303_0 conda-forge
tensorflow-addons 0.14.0 py39hfe13ede_0 <unknown>
tensorflow-base 2.6.2 cpu_py39hbcb9a37_0 conda-forge
tensorflow-estimator 2.6.2 cpu_py39h1b7c303_0 conda-forge
termcolor 1.1.0 py_2 conda-forge
threadpoolctl 3.0.0 pyh8a188c0_0 conda-forge
tifffile 2021.11.2 pyhd8ed1ab_0 conda-forge
tk 8.6.11 h27826a3_1 conda-forge
toml 0.10.2 pyhd8ed1ab_0 conda-forge
toolz 0.11.2 pyhd8ed1ab_0 conda-forge
tqdm 4.62.3 pyhd8ed1ab_0 conda-forge
typeguard 2.13.0 pyhd8ed1ab_0 conda-forge
typing-extensions 3.7.4.3 0 conda-forge
typing_extensions 3.7.4.3 py_0 conda-forge
tzdata 2021e he74cb21_0 conda-forge
urllib3 1.26.7 pyhd8ed1ab_0 conda-forge
werkzeug 2.0.1 pyhd8ed1ab_0 conda-forge
wheel 0.37.0 pyhd8ed1ab_1 conda-forge
wrapt 1.12.1 py39h3811e60_3 conda-forge
xz 5.2.5 h516909a_1 conda-forge
yaml 0.2.5 h516909a_0 conda-forge
yarl 1.7.2 py39h3811e60_1 conda-forge
zfp 0.5.5 h9c3ff4c_8 conda-forge
zipp 3.6.0 pyhd8ed1ab_0 conda-forge
zlib 1.2.11 h36c2ea0_1013 conda-forge
zstd 1.5.0 ha95c52a_0 conda-forge
Hi looking through this issue I was able to reproduce the error @h-vetinari ran into with the following python script:
from _pytest.config import get_config
cfg = get_config()
# This does not work
cfg._parser.addoption(
"--deadbeef",
action="store_true"
)
cfg.getoption("--deadbeef") # value error thrown here
while changing the arguments passed to addoptions to @h-vetinari workaround fixes things:
from _pytest.config import get_config
cfg = get_config()
# This works
cfg._parser.addoption(
"--livebeef",
action="store",
default=True
)
cfg.getoption("--livebeef")
I'm wondering if the reason @yuvalshi0 wasn't able to reproduce this error is because pytester.parseconfig or pytester.makeconftest is patching the bug.
I want to look further into this issue and work on a fix for it, just don't want to step on anyone's toes if they're already working on a fix.
Hi looking through this issue I was able to reproduce the error @h-vetinari ran into with the following python script:
from _pytest.config import get_config cfg = get_config() # This does not work cfg._parser.addoption( "--deadbeef", action="store_true" ) cfg.getoption("--deadbeef") # value error thrown herewhile changing the arguments passed to addoptions to @h-vetinari workaround fixes things:
from _pytest.config import get_config cfg = get_config() # This works cfg._parser.addoption( "--livebeef", action="store", default=True ) cfg.getoption("--livebeef")I'm wondering if the reason @yuvalshi0 wasn't able to reproduce this error is because
pytester.parseconfigorpytester.makeconftestis patching the bug. I want to look further into this issue and work on a fix for it, just don't want to step on anyone's toes if they're already working on a fix.
Go ahead ;)
please note that pytest uses a internal api to compose the argument parser later
one should never use cfg._parser.addoption like the last snippet
@RonnyPfannschmidt thanks for the feedback, will work on creating a proper test for reproducing the error instead of the hacky snippet I used. Creating a proper reproduction of the error should also help in finding/developing a fix.
So it looks like after doing a proper test, the test passes and the getoption method no longer throws a value error, my motivation for doing the tests this way was to try and mimic the way in which the tf_addons are being tested:
File Structure:
test_like_tf_addons
-- __init__.py
-- conftest.py
-- test_it.py
-- utils.py
File Contents:
# conftest.py
import pytest
from .utils import pytest_addoption, some_fixture
# test_it.py
import pytest
def test_it_already():
assert True
# utils.py
import pytest
def pytest_addoption(parser):
parser.addoption(
"--dead-beef",
action="store_true"
)
@pytest.fixture(scope="session", autouse=True)
def some_fixture(request):
val = request.config.getoption("--dead-beef")
I also made sure to test things with the same versions used in the failed build link and here are the results:
platform linux -- Python 3.9.7, pytest-6.2.5, py-1.11.0, pluggy-1.0.0 -- /home/ksantana/sandbox/oss/pytest-play-plugin/.venv/bin/python
cachedir: .pytest_cache
rootdir: /home/ksantana/sandbox/oss/pytest-play-plugin/test_like_tf_addons
plugins: typeguard-2.13.0
collected 1 item
test_it.py::test_it_already PASSED
I've also tested pulling down the tf_addons project, running the commands from here link, then running the tests and all the tests still seem to pass. Motivation here was to see if maybe it was an issue with the build process.
Still looking into this, just wanted to post an update of the things I've tried and see if someone has other ideas on what could be the problem.
@kevinsantana11 it may be helpfull to put the conftest into a subdirectory
i suspect the issue may be related to conftest discovery and initial conftest loading
for ensuring the rootdir, adding a pytest.ini or a setup.cfg may help
@RonnyPfannschmidt it looks like you're recommendation was very helpful, thank you very much!. So after digging into the code these are my thoughts:
Issue occurs because the initial pass that collects the conftests does not look in directories that don't begin with test* as we see on _pytest.config.PytestPluginManager._try_load_conftest:511. Subsequently if the conftests isn't picked up in that stage the options it adds won't be parsed during the parse_setoption process. We see this become a problem in the tensorflow addons project because the conftest is embedded in the addons folder, running pytest from the root level dir will cause the value error we are seeing. In the previous situation the conftest is later picked up during the collection round right before the test round starts.
my question now is where should the patch live, should I fix this by extending _try_load_conftests to search sub directories that do not start with test*or parse the arguments picked up after the collection round.