pybind11 Demo: Testing pybind11-stubgen + mypy

Description

This PR shows how pybind11-stubgen and mypy could be used in pytest to check for correct type hints. It stems from the discussion at #5663.

This is just a quick demo with one positive and one negative test.

Sources for calling pybind11-stubgen and mypy from Python:

Integrating mypy into another Python application
Calling pybind11-stubgen from Python: sizmailov/pybind11-stubgen#252 (thx @gentlegiantJGC for this 👍)

TODOs:

[ ] Integrate test cases from #4888
[x] Maybe use this to generate stubs for all test modules (just to check if pybind11-stubgen and mypy run successful or find errors)
[ ] Pin versions of mypy and pybind11-stubgen (or even use a version matrix in CI).
[ ] Consider other type checkers (most notably Pyright/Pylance since this is the default in vscode).

📚 Documentation preview 📚: https://pybind11--5678.org.readthedocs.build/

May 20 '25 22:05 timohl

Some errors I see so far:

Python 3.14 seems like causing some mypy internal errors
Some checks do not properly install pybind11-stubgen, although I added it to requirements.txt and pyproject.toml
mypy report on https://github.com/pybind/pybind11/actions/runs/15149485803/job/42592785722?pr=5678 seems scrambled (maybe some formatting or windows specific stuff)

May 20 '25 23:05 timohl

The new test dependencies seem "small/lightweight" and "super lightweight":

$ time pip install mypy
Collecting mypy
  Using cached mypy-1.15.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_28_x86_64.whl.metadata (2.1 kB)
Requirement already satisfied: typing_extensions>=4.6.0 in ./python/venvs/scratch/cpwheels/lib/python3.12/site-packages (from mypy) (4.13.2)
Requirement already satisfied: mypy_extensions>=1.0.0 in ./python/venvs/scratch/cpwheels/lib/python3.12/site-packages (from mypy) (1.1.0)
Using cached mypy-1.15.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_28_x86_64.whl (12.4 MB)
Installing collected packages: mypy
Successfully installed mypy-1.15.0

real    0m1.098s
user    0m0.936s
sys     0m0.150s

$ time pip install pybind11-stubgen
Collecting pybind11-stubgen
  Using cached pybind11_stubgen-2.5.4-py3-none-any.whl.metadata (1.9 kB)
Using cached pybind11_stubgen-2.5.4-py3-none-any.whl (30 kB)
Installing collected packages: pybind11-stubgen
Successfully installed pybind11-stubgen-2.5.4

real    0m0.418s
user    0m0.242s
sys     0m0.048s

I think you're going in a great direction.

Maybe use this to generate stubs for all test modules (just to check if pybind11-stubgen and mypy run successful or find errors)

That sounds like a great idea, as long as you build in an easy way to turn that off during development iterations. I'd only want to turn that on again while I clean up for the next git push.

If you do that, I'd say forget the tests under #4888 (those came mostly from mypy). I believe they'll be 100% redundant.

Pyright/Pylance since this is the default in vscode

I'd look at that is the top priority. — I don't know a lot about typing, but last time I quizzed ChatGPT about it (a few months ago), it told me mypy is a good start, but — liberally reinterpreted by me — what really counts is LSP.

May 21 '25 04:05 rwgk

If you haven't started using uv, you should try it, even if it's just uv venv and uv pip:

$ uv venv
Using CPython 3.13.3 interpreter at: /usr/local/opt/[email protected]/bin/python3.13
Creating virtual environment at: .venv
Activate with: source .venv/bin/activate.fish

________________________________________________________
Executed in   28.56 millis    fish           external
   usr time   12.44 millis    0.25 millis   12.19 millis
   sys time   13.04 millis    1.91 millis   11.13 millis

$ time uv pip install mypy
Resolved 3 packages in 3ms
Installed 3 packages in 50ms
 + mypy==1.15.0
 + mypy-extensions==1.1.0
 + typing-extensions==4.13.2

________________________________________________________
Executed in   81.80 millis    fish           external
   usr time   18.27 millis    0.29 millis   17.98 millis
   sys time   54.27 millis    3.55 millis   50.73 millis

I think ty and pyrefly, the two new Python type checkers in Rust, are likely to have a big impact in the future, but for now, most projects work on passing mypy, and the existing type checkers like pylance know that and try to be compatible. I don't think I use anything other than mypy in CI on all my projects, though I use pyright locally in VSCode and via LSP in Vim.

The bug in Python 3.14.0b1 that we've reported also affects mypy, and it's been marked as a release blocker. So mypy will likely work better in beta 2.

May 21 '25 15:05 henryiii

I fixed most of the CI issues:

Added xfail for Python 3.14
Using --no-color-output fixes parsing the mypy report correctly
I have changed CI at a couple of places to use tests/requirements.txt instead of manually installing pytest, numpy and so on. Since I added pybind11-stubgen and mypy there, the tests now run successfully (except for mingw because I am not sure how to change this the right was)

@henryiii The CI is uses a mix of uv, pip, pipx, apt and the github actions setup-python and setup-uv. Would it make sense to have this more consistently use uv, setup-python and setup-uv where possible (mingw seems to be an exception)? Also is there a downside to using venv in CI checks? In https://github.com/pybind/pybind11/actions/runs/15182964725/job/42696579100 uv complained about the system-wide environment being externally managed (while almost all other jobs did not complain). Adding activate-environment: true to the setup-uv action solved this.

@rwgk

Maybe use this to generate stubs for all test modules (just to check if pybind11-stubgen and mypy run successful or find errors)

That sounds like a great idea, as long as you build in an easy way to turn that off during development iterations. I'd only want to turn that on again while I clean up for the next git push.

Probably best to use pytest markers for that (which could be excluded by default). Custom markers (pytest docs) Exclude custom marker by default (stackoverflow)

May 22 '25 11:05 timohl

I have added a test that applies stubgen and mypy to the whole pybind11_tests module.

This is the stubgen error log:

error log


ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.lvalue_nested : Can't find/import 'RValueCaster'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.lvalue_pair : Can't find/import 'RValueCaster'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.lvalue_tuple : Can't find/import 'RValueCaster'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.nodefer_none_void : Can't find/import 'types.CapsuleType'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.rvalue_nested : Can't find/import 'RValueCaster'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.rvalue_pair : Can't find/import 'RValueCaster'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.rvalue_tuple : Can't find/import 'RValueCaster'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.takes : Can't find/import 'ConstRefCasted'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.takes_const_ptr : Can't find/import 'ConstRefCasted'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.takes_const_ref : Can't find/import 'ConstRefCasted'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.takes_const_ref_wrap : Can't find/import 'ConstRefCasted'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.takes_move : Can't find/import 'ConstRefCasted'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.takes_ptr : Can't find/import 'ConstRefCasted'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.takes_ref : Can't find/import 'ConstRefCasted'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.takes_ref_wrap : Can't find/import 'ConstRefCasted'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.callbacks : Can't find/import 'pybind11_tests.callbacks.custom_function'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.callbacks : Can't find/import 'pybind11_tests.callbacks.custom_function2'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.callbacks.func_accepting_func_accepting_base : Invalid expression 'test_submodule_callbacks(module_&)::AbstractBase'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.class_.ProtectedB.get_self : Can't find/import 'types.CapsuleType'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.class_.ProtectedB.void_foo : Can't find/import 'types.CapsuleType'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.class_ : Invalid identifier ''
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.local_bindings.get_mixed_gl : Invalid expression 'LocalBase'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.local_bindings.get_mixed_lg : Invalid expression 'LocalBase'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.local_bindings.load_external1 : Invalid expression 'LocalBase'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.local_bindings.load_external2 : Invalid expression 'LocalBase'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.methods_and_attributes.exercise_is_setter.Field.int_value. : Invalid expression 'pybind11_tests::exercise_is_setter::FieldBase'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.native_enum.pass_some_proto_enum : Invalid expression 'test_native_enum::some_proto_enum'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.native_enum.return_some_proto_enum : Invalid expression 'test_native_enum::some_proto_enum'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_array.index_using_ellipsis : Invalid expression 'detail::accessor<:accessor_policies::generic_item>'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_dtypes.create_array_array : Can't find/import 'ArrayStruct'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_dtypes.create_complex_array : Can't find/import 'ComplexStruct'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_dtypes.create_enum_array : Can't find/import 'EnumStruct'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_dtypes.create_rec_nested : Can't find/import 'NestedStruct'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_dtypes.create_rec_partial : Can't find/import 'PartialStruct'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_dtypes.create_rec_partial_nested : Can't find/import 'PartialNestedStruct'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_dtypes.create_string_array : Can't find/import 'StringStruct'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_dtypes.f_nested : Can't find/import 'NestedStruct'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_dtypes.print_array_array : Can't find/import 'ArrayStruct'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_dtypes.print_complex_array : Can't find/import 'ComplexStruct'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_dtypes.print_enum_array : Can't find/import 'EnumStruct'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_dtypes.print_rec_nested : Can't find/import 'NestedStruct'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_dtypes.print_string_array : Can't find/import 'StringStruct'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.opaque_types.get_void_ptr_value : Can't find/import 'types.CapsuleType'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.opaque_types.return_void_ptr : Can't find/import 'types.CapsuleType'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.annotate_never : Can't find/import 'typing.Never'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.annotate_type_is : Can't find/import 'typing.TypeIs'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.check_type_is : Can't find/import 'typing.TypeIs'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.return_capsule_with_destructor : Can't find/import 'types.CapsuleType'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.return_capsule_with_destructor_2 : Can't find/import 'types.CapsuleType'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.return_capsule_with_destructor_3 : Can't find/import 'types.CapsuleType'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.return_capsule_with_explicit_nullptr_dtor : Can't find/import 'types.CapsuleType'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.return_capsule_with_name_and_destructor : Can't find/import 'types.CapsuleType'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.return_renamed_capsule_with_destructor : Can't find/import 'types.CapsuleType'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.return_renamed_capsule_with_destructor_2 : Can't find/import 'types.CapsuleType'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.test_list_slicing : Invalid expression 'detail::accessor<:accessor_policies::generic_item>'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.test_list_slicing_default : Invalid expression 'detail::accessor<:accessor_policies::generic_item>'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.weakref_from_handle : Can't find/import 'weakref'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.weakref_from_handle_and_function : Can't find/import 'weakref'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.weakref_from_object : Can't find/import 'weakref'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.weakref_from_object_and_function : Can't find/import 'weakref'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.smart_ptr.ContainerUsingPrivateESFT.ptr. : Invalid expression 'test_submodule_smart_ptr(module_&)::PrivateESFT'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.smart_ptr.ContainerUsingPrivateESFT.ptr. : Invalid expression 'test_submodule_smart_ptr(module_&)::PrivateESFT'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.smart_ptr.return_std_shared_ptr_example_drvd : Invalid expression 'holder_caster_traits_test::example_drvd'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.smart_ptr.return_std_unique_ptr_example_drvd : Invalid expression 'holder_caster_traits_test::example_drvd'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.cast_lv_array : Can't find/import 'RValueCaster'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.cast_lv_map : Can't find/import 'RValueCaster'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.cast_lv_nested : Can't find/import 'RValueCaster'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.cast_lv_vector : Can't find/import 'RValueCaster'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.cast_ptr_vector : Can't find/import 'RValueCaster'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.cast_rv_array : Can't find/import 'RValueCaster'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.cast_rv_map : Can't find/import 'RValueCaster'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.cast_rv_nested : Can't find/import 'RValueCaster'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.cast_rv_vector : Can't find/import 'RValueCaster'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.func_with_string_or_vector_string_arg_overload : Invalid expression 'std::vector<:__cxx11::basic_string error pybind11_stubgen:error_handlers.py:77 in pybind11_tests.stl.func_with_string_or_vector_string_arg_overload : invalid expression>'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.func_with_string_or_vector_string_arg_overload : Invalid expression ':allocator >'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.func_with_string_or_vector_string_arg_overload : Invalid expression ':allocator<:__cxx11::basic_string error pybind11_stubgen:error_handlers.py:77 in pybind11_tests.stl.func_with_string_or_vector_string_arg_overload : invalid expression> > > >'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.tpl_constr_optional : Can't find/import 'TplCtorClass'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.tpl_ctor_map : Can't find/import 'TplCtorClass'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.tpl_ctor_set : Can't find/import 'TplCtorClass'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.tpl_ctor_vector : Can't find/import 'TplCtorClass'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stubgen_error.identity_capsule : Can't find/import 'types.CapsuleType'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.virtual_functions.Adder.__call__ : Invalid expression 'AdderBase::Data'
WARNING  pybind11_stubgen:error_handlers.py:150 Raw C++ types/values were found in signatures extracted from docstrings.
Please check the corresponding sections of pybind11 documentation to avoid common mistakes in binding code:
 - https://pybind11.readthedocs.io/en/latest/advanced/misc.html#avoiding-cpp-types-in-docstrings
 - https://pybind11.readthedocs.io/en/latest/advanced/functions.html#default-arguments-revisited

This is the mypy report:

/tmp/pytest-of-timohl/pytest-5/test_stubgen_all0/pybind11_tests/stl.pyi:187: error: Ellipses cannot accompany other argument types in function type signature  [syntax]
/tmp/pytest-of-timohl/pytest-5/test_stubgen_all0/pybind11_tests/stl.pyi:187: error: Duplicate argument "std" in function definition
Found 2 errors in 1 file (errors prevented further checking)

I have extracted __doc__ of the function mentioned by mypy to see, what it actually contains:

func_with_string_or_vector_string_arg_overload(*args, **kwargs)
Overloaded function.

1. func_with_string_or_vector_string_arg_overload(arg0: std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >) -> int

2. func_with_string_or_vector_string_arg_overload(arg0: collections.abc.Sequence[str]) -> int

3. func_with_string_or_vector_string_arg_overload(arg0: str) -> int

The function is bound here: https://github.com/pybind/pybind11/blob/98bd78f063b2f30570740030cb2d13b2a62a062c/tests/test_stl.cpp#L577-L582

Not sure what is going on here. I would have assumed that the list_caster would give collections.abc.Sequence: https://github.com/pybind/pybind11/blob/98bd78f063b2f30570740030cb2d13b2a62a062c/include/pybind11/stl.h#L360-L366

I will go through the other errors tomorrow.

May 23 '25 17:05 timohl

I have added a marker -m stubgen to enable testing of stubgen on all test modules (off by default right now). In order to enable this test, you can use cmake with:

cmake -S . -B build -DPYBIND11_WERROR=ON -DDOWNLOAD_CATCH=ON -DDOWNLOAD_EIGEN=ON -DPYBIND11_PYTEST_ARGS="-m stubgen"

Also, I went through some of the stubgen errors:

The following stubgen errors were straigt forward to fix by changing the name attribute of the type_caster:

error log


ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.lvalue_nested : Can't find/import 'RValueCaster'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.lvalue_pair : Can't find/import 'RValueCaster'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.lvalue_tuple : Can't find/import 'RValueCaster'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.nodefer_none_void : Can't find/import 'types.CapsuleType'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.rvalue_nested : Can't find/import 'RValueCaster'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.rvalue_pair : Can't find/import 'RValueCaster'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.rvalue_tuple : Can't find/import 'RValueCaster'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.takes : Can't find/import 'ConstRefCasted'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.takes_const_ptr : Can't find/import 'ConstRefCasted'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.takes_const_ref : Can't find/import 'ConstRefCasted'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.takes_const_ref_wrap : Can't find/import 'ConstRefCasted'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.takes_move : Can't find/import 'ConstRefCasted'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.takes_ptr : Can't find/import 'ConstRefCasted'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.takes_ref : Can't find/import 'ConstRefCasted'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.takes_ref_wrap : Can't find/import 'ConstRefCasted'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.cast_lv_array : Can't find/import 'RValueCaster'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.cast_lv_map : Can't find/import 'RValueCaster'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.cast_lv_nested : Can't find/import 'RValueCaster'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.cast_lv_vector : Can't find/import 'RValueCaster'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.cast_ptr_vector : Can't find/import 'RValueCaster'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.cast_rv_array : Can't find/import 'RValueCaster'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.cast_rv_map : Can't find/import 'RValueCaster'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.cast_rv_nested : Can't find/import 'RValueCaster'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.cast_rv_vector : Can't find/import 'RValueCaster'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.copy_move_policies.copy_only : Can't find/import 'CopyOnlyInt'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.copy_move_policies.copy_optional : Can't find/import 'CopyOnlyInt'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.copy_move_policies.copy_tuple : Can't find/import 'CopyOnlyInt'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.copy_move_policies.move_copy_nested : Can't find/import 'MoveOnlyInt'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.copy_move_policies.move_copy_nested : Can't find/import 'MoveOrCopyInt'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.copy_move_policies.move_copy_nested : Can't find/import 'CopyOnlyInt'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.copy_move_policies.move_only : Can't find/import 'MoveOnlyInt'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.copy_move_policies.move_optional : Can't find/import 'MoveOnlyInt'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.copy_move_policies.move_optional_tuple : Can't find/import 'MoveOrCopyInt'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.copy_move_policies.move_optional_tuple : Can't find/import 'MoveOnlyInt'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.copy_move_policies.move_optional_tuple : Can't find/import 'CopyOnlyInt'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.copy_move_policies.move_or_copy : Can't find/import 'MoveOrCopyInt'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.copy_move_policies.move_or_copy_optional : Can't find/import 'MoveOrCopyInt'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.copy_move_policies.move_pair : Can't find/import 'MoveOnlyInt'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.copy_move_policies.move_pair : Can't find/import 'MoveOrCopyInt'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.copy_move_policies.move_tuple : Can't find/import 'MoveOnlyInt'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.copy_move_policies.move_tuple : Can't find/import 'MoveOrCopyInt'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.custom_type_casters.ArgInspector.f : Can't find/import 'ArgInspector1'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.custom_type_casters.ArgInspector.f : Can't find/import 'ArgAlwaysConverts'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.custom_type_casters.ArgInspector.g : Can't find/import 'ArgInspector1'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.custom_type_casters.ArgInspector.g : Can't find/import 'ArgInspector2'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.custom_type_casters.ArgInspector.g : Can't find/import 'ArgAlwaysConverts'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.custom_type_casters.ArgInspector.h : Can't find/import 'ArgInspector2'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.custom_type_casters.ArgInspector.h : Can't find/import 'ArgAlwaysConverts'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.custom_type_casters.arg_inspect_func : Can't find/import 'ArgInspector2'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.custom_type_casters.arg_inspect_func : Can't find/import 'ArgInspector1'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.custom_type_casters.arg_inspect_func : Can't find/import 'ArgAlwaysConverts'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.custom_type_casters.custom_caster_destroy : Can't find/import 'DestructionTester'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.custom_type_casters.custom_caster_destroy_const : Can't find/import 'DestructionTester'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.custom_type_casters.custom_caster_no_destroy : Can't find/import 'DestructionTester'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.custom_type_casters.other_lib_type : Can't find/import 'MyType'

Also the following stubgen errors should be fixed by #5693.

error log


ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.class_.ProtectedB.get_self : Can't find/import 'types.CapsuleType'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.class_.ProtectedB.void_foo : Can't find/import 'types.CapsuleType'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.class_.pointers_equal : Can't find/import 'types.CapsuleType'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.class_.read_foo : Can't find/import 'types.CapsuleType'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.opaque_types.get_void_ptr_value : Can't find/import 'types.CapsuleType'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.opaque_types.return_void_ptr : Can't find/import 'types.CapsuleType'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.return_capsule_with_destructor : Can't find/import 'types.CapsuleType'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.return_capsule_with_destructor_2 : Can't find/import 'types.CapsuleType'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.return_capsule_with_destructor_3 : Can't find/import 'types.CapsuleType'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.return_capsule_with_explicit_nullptr_dtor : Can't find/import 'types.CapsuleType'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.return_capsule_with_name_and_destructor : Can't find/import 'types.CapsuleType'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.return_renamed_capsule_with_destructor : Can't find/import 'types.CapsuleType'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.return_renamed_capsule_with_destructor_2 : Can't find/import 'types.CapsuleType'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stubgen_error.identity_capsule : Can't find/import 'types.CapsuleType'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.annotate_never : Can't find/import 'typing.Never'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.annotate_type_is : Can't find/import 'typing.TypeIs'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.check_type_is : Can't find/import 'typing.TypeIs'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.weakref_from_handle : Can't find/import 'weakref'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.weakref_from_handle_and_function : Can't find/import 'weakref'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.weakref_from_object : Can't find/import 'weakref'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.weakref_from_object_and_function : Can't find/import 'weakref'

The following stubgen errors I did not look at or I did not see an easy fix:

error log


ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.callbacks : Can't find/import 'pybind11_tests.callbacks.custom_function'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.callbacks : Can't find/import 'pybind11_tests.callbacks.custom_function2'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.callbacks.func_accepting_func_accepting_base : Invalid expression 'test_submodule_callbacks(module_&)::AbstractBase'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.class_ : Invalid identifier ''
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.local_bindings.get_mixed_gl : Invalid expression 'LocalBase'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.local_bindings.get_mixed_lg : Invalid expression 'LocalBase'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.local_bindings.load_external1 : Invalid expression 'LocalBase'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.local_bindings.load_external2 : Invalid expression 'LocalBase'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.methods_and_attributes.exercise_is_setter.Field.int_value. : Invalid expression 'pybind11_tests::exercise_is_setter::FieldBase'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.native_enum.pass_some_proto_enum : Invalid expression 'test_native_enum::some_proto_enum'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.native_enum.return_some_proto_enum : Invalid expression 'test_native_enum::some_proto_enum'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_array.index_using_ellipsis : Invalid expression 'detail::accessor<:accessor_policies::generic_item>'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_dtypes.create_array_array : Can't find/import 'ArrayStruct'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_dtypes.create_complex_array : Can't find/import 'ComplexStruct'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_dtypes.create_enum_array : Can't find/import 'EnumStruct'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_dtypes.create_rec_nested : Can't find/import 'NestedStruct'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_dtypes.create_rec_partial : Can't find/import 'PartialStruct'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_dtypes.create_rec_partial_nested : Can't find/import 'PartialNestedStruct'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_dtypes.create_string_array : Can't find/import 'StringStruct'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_dtypes.f_nested : Can't find/import 'NestedStruct'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_dtypes.print_array_array : Can't find/import 'ArrayStruct'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_dtypes.print_complex_array : Can't find/import 'ComplexStruct'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_dtypes.print_enum_array : Can't find/import 'EnumStruct'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_dtypes.print_rec_nested : Can't find/import 'NestedStruct'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_dtypes.print_string_array : Can't find/import 'StringStruct'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.test_list_slicing : Invalid expression 'detail::accessor<:accessor_policies::generic_item>'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.test_list_slicing_default : Invalid expression 'detail::accessor<:accessor_policies::generic_item>'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.smart_ptr.ContainerUsingPrivateESFT.ptr. : Invalid expression 'test_submodule_smart_ptr(module_&)::PrivateESFT'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.smart_ptr.ContainerUsingPrivateESFT.ptr. : Invalid expression 'test_submodule_smart_ptr(module_&)::PrivateESFT'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.smart_ptr.return_std_shared_ptr_example_drvd : Invalid expression 'holder_caster_traits_test::example_drvd'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.smart_ptr.return_std_unique_ptr_example_drvd : Invalid expression 'holder_caster_traits_test::example_drvd'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.func_with_string_or_vector_string_arg_overload : Invalid expression 'std::vector<:__cxx11::basic_string error pybind11_stubgen:error_handlers.py:77 in pybind11_tests.stl.func_with_string_or_vector_string_arg_overload : invalid expression>'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.func_with_string_or_vector_string_arg_overload : Invalid expression ':allocator >'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.func_with_string_or_vector_string_arg_overload : Invalid expression ':allocator<:__cxx11::basic_string error pybind11_stubgen:error_handlers.py:77 in pybind11_tests.stl.func_with_string_or_vector_string_arg_overload : invalid expression> > > >'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.tpl_constr_optional : Can't find/import 'TplCtorClass'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.tpl_ctor_map : Can't find/import 'TplCtorClass'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.tpl_ctor_set : Can't find/import 'TplCtorClass'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.tpl_ctor_vector : Can't find/import 'TplCtorClass'
ERROR    pybind11_stubgen:error_handlers.py:77 In pybind11_tests.virtual_functions.Adder.__call__ : Invalid expression 'AdderBase::Data'

The error I described in my previous comment from func_with_string_or_vector_string_arg_overload stems from: https://github.com/pybind/pybind11/blob/03b4a9e56fc0e8f8c8edd0f89dabd91113063cb8/tests/test_stl.cpp#L63 String vectors are made opaque here but are not bound to a Python type (see "Making opaque types" docs

May 25 '25 23:05 timohl

Stubgen tests are now just enabled on the Ubuntu jobs (could be discussed where to roll out stubgen testing in the end).

I merged the fixes from #5693 into this (probably have to do a rebase later to fix the git history, but I wanted to filter out the errors fixed by this).

Also some test binding fixes in f9e4686, but the change to numpy dtype tests breaks now (gotta understand this test more)

May 28 '25 16:05 timohl

Just a quick update on this: I am pretty busy right now with finishing my master thesis, so I won't have much time to work on this until mid next month.

One question though: Would it make sense to have this as a smaller PR adding stubgen/mypy tests that are disabled by default but can be manually activated with something like -DPYBIND11_PYTEST_ARGS="-m stubgen" and not including pybind11-stubgen and mypy to the test requirements yet. That way, people could already use this for development. Fixing/handling all stubgen/mypy errors in the test code, as well as integration into the CI workflows, could then maybe go into a subsequent PR?

Jun 25 '25 08:06 timohl

Just a quick update on this: I am pretty busy right now with finishing my master thesis, so I won't have much time to work on this until mid next month.

One question though: Would it make sense to have this as a smaller PR adding stubgen/mypy tests that are disabled by default but can be manually activated with something like -DPYBIND11_PYTEST_ARGS="-m stubgen" and not including pybind11-stubgen and mypy to the test requirements yet. That way, people could already use this for development. Fixing/handling all stubgen/mypy errors in the test code, as well as integration into the CI workflows, could then maybe go into a subsequent PR?

That sounds good to me, although maybe that'll be best a couple days after the pybind11 v3.0.0 release? @henryiii

Jun 25 '25 14:06 rwgk

Now that pybind11 3.0.0 is out and if you have the free time. I would love the -m stubgen feature. It makes testing new typing stuff much easier.

Jul 12 '25 21:07 InvincibleRMC