typeshed icon indicating copy to clipboard operation
typeshed copied to clipboard
verified publisher icon python

Adding stubs for `ruamel.yaml`

Open stevenlele opened this issue 1 year ago • 13 comments

ruamel.yaml is a fork of PyYAML which features a new API and can perform round-trip conversion preserving the original format and comments. It also claims to have fixed various issues when the development of PyYAML was inactive.

Typeshed testing depends on this package via pre-commit-hooks.

  • PyPI: https://pypi.org/project/ruamel.yaml/
  • Documentation: https://yaml.readthedocs.io/en/latest/
  • PyYAML documentation: https://pyyaml.org/wiki/PyYAMLDocumentation
  • YAML spec: https://yaml.org/spec/1.2.2/
Repository SourceForge GitHub mirror
ruamel.yaml https://sourceforge.net/p/ruamel-yaml/ https://github.com/commx/ruamel-yaml
ruamel.yaml.clib https://sourceforge.net/p/ruamel-yaml-clib/ https://github.com/ruamel/yaml.clib
Test data https://sourceforge.net/p/ruamel-yaml-data/

Although the package ships a py.typed file, almost every type hint is Any, which makes coding experience terrible. This falls into the situation mentioned in https://github.com/python/typeshed/issues/11955, which might need some discussion.

The typing information in the package seems to originate from https://sourceforge.net/p/ruamel-yaml/tickets/42/ (from https://github.com/common-workflow-language/schema_salad) in 2016 when things were still in Python 2. Then it was never properly maintained and became a pile of Anys.

This package doesn't follow the best practice of prefixing private properties and methods with underscore(s), which makes writing stubs a lot more complex due to the large number of "public" properties and methods.

It's also worth noting that the documentation isn't well-structured, nor does it cover necessary information like every format option, which makes it even harder to use this library. Adding proper typing information might provide some help.

~~Since there are already stubs for PyYAML, many modules of ruamel.yaml can be added by referring to the corresponding ones in PyYAML.~~ They turned out to be not very helpful.

A special case in this library is the YAML class, which you need to pass a typ argument to select its parsing mode. Depending on the typ, the methods of YAML behave differently. For example, YAML(typ='rt').seq() returns a CommentedSeq with round-trip-specific methods, while YAML(typ='safe') returns a plain list. To make coding easier, I added pseudo-subclasses of YAML (like _RoundTripYAML) and converted the __init__() method into multiple @overload def __new__() methods which returns different variants based on the typ parameter.

There's also an undocumented rtsc (round-trip split comments) type in the library, but the parsers seem to be very experimental and less-maintained, so I intentionally omitted it in the main module. I found the only usage at https://github.com/SoulMelody/LibreSVIP/blob/main/libresvip/utils/yamlutils/init.py#L15 which doesn't seem to be intentional or necessary. Edit: The parser doesn't seem to be working at all on a YAML document with comments.

Just for reference, I found the one and only ruamel.yaml plug-in code example at https://github.com/dstl/Stone-Soup/blob/main/stonesoup/serialise.py which might help understanding how that plug_in parameter works.

MonkeyType provides great help.

stevenlele avatar Aug 24 '24 12:08 stevenlele

Diff from mypy_primer, showing the effect of this PR on open source code:

mitmproxy (https://github.com/mitmproxy/mitmproxy)
+ mitmproxy/optmanager.py:496: error: Argument 1 to "dump" of "YAML" has incompatible type "CommentedMap"; expected "Path | str | bytes | SupportsRead[str | bytes]"  [arg-type]

paasta (https://github.com/yelp/paasta)
+ paasta_tools/cli/cmds/validate.py:44: error: Module "ruamel.yaml" has no attribute "SafeConstructor"  [attr-defined]
+ paasta_tools/cli/cmds/validate.py:339: error: "Type[BaseConstructor]" has no attribute "flatten_mapping"  [attr-defined]

check-jsonschema (https://github.com/python-jsonschema/check-jsonschema)
+ src/check_jsonschema/parsers/yaml.py:8: error: Module has no attribute "YAMLError"  [attr-defined]
+ src/check_jsonschema/parsers/yaml.py:18: error: Name "ruamel.yaml.SafeConstructor" is not defined  [name-defined]
+ src/check_jsonschema/parsers/yaml.py:27: error: Constructor? has no attribute "yaml_constructors"  [attr-defined]
+ src/check_jsonschema/parsers/yaml.py:28: error: Constructor? has no attribute "yaml_constructors"  [attr-defined]
+ src/check_jsonschema/parsers/yaml.py:59: error: Name "ruamel.yaml.YAMLError" is not defined  [name-defined]
+ src/check_jsonschema/parsers/yaml.py:66: error: Module has no attribute "YAMLError"  [attr-defined]
+ src/check_jsonschema/transforms/gitlab.py:22: error: Name "ruamel.yaml.BaseConstructor" is not defined  [name-defined]

github-actions[bot] avatar Aug 24 '24 12:08 github-actions[bot]

Diff from mypy_primer, showing the effect of this PR on open source code:

mitmproxy (https://github.com/mitmproxy/mitmproxy)
+ mitmproxy/optmanager.py:482: error: Need type annotation for "s"  [var-annotated]
+ mitmproxy/optmanager.py:496: error: Argument 1 to "dump" of "YAML" has incompatible type "CommentedMap[Any, Any]"; expected "Path | SupportsWrite[str | bytes]"  [arg-type]
+ mitmproxy/tools/console/keymap.py:239: error: Item "StreamMark" of "Mark | StreamMark | None" has no attribute "get_snippet"  [union-attr]
+ mitmproxy/tools/console/keymap.py:239: error: Item "None" of "Mark | StreamMark | None" has no attribute "get_snippet"  [union-attr]
+ mitmproxy/tools/console/keymap.py:242: error: Item "None" of "Mark | StreamMark | None" has no attribute "line"  [union-attr]

paasta (https://github.com/yelp/paasta)
+ paasta_tools/cli/cmds/validate.py:44: error: Module "ruamel.yaml" has no attribute "SafeConstructor"  [attr-defined]
+ paasta_tools/cli/cmds/validate.py:339: error: "Type[BaseConstructor]" has no attribute "flatten_mapping"  [attr-defined]

spack (https://github.com/spack/spack)
+ lib/spack/spack/util/spack_yaml.py:201: error: Cannot infer type argument 1 of "add_representer" of "BaseRepresenter"  [misc]
+ lib/spack/spack/util/spack_yaml.py:204: error: Cannot infer type argument 1 of "add_representer" of "BaseRepresenter"  [misc]

check-jsonschema (https://github.com/python-jsonschema/check-jsonschema)
+ src/check_jsonschema/parsers/yaml.py:8: error: Module has no attribute "YAMLError"  [attr-defined]
+ src/check_jsonschema/parsers/yaml.py:18: error: Name "ruamel.yaml.SafeConstructor" is not defined  [name-defined]
+ src/check_jsonschema/parsers/yaml.py:27: error: Constructor? has no attribute "yaml_constructors"  [attr-defined]
+ src/check_jsonschema/parsers/yaml.py:28: error: Constructor? has no attribute "yaml_constructors"  [attr-defined]
+ src/check_jsonschema/parsers/yaml.py:59: error: Name "ruamel.yaml.YAMLError" is not defined  [name-defined]
+ src/check_jsonschema/parsers/yaml.py:66: error: Module has no attribute "YAMLError"  [attr-defined]
+ src/check_jsonschema/transforms/gitlab.py:22: error: Name "ruamel.yaml.BaseConstructor" is not defined  [name-defined]

github-actions[bot] avatar Sep 12 '24 19:09 github-actions[bot]

Diff from mypy_primer, showing the effect of this PR on open source code:

mitmproxy (https://github.com/mitmproxy/mitmproxy)
+ mitmproxy/optmanager.py:482: error: Need type annotation for "s"  [var-annotated]
+ mitmproxy/optmanager.py:496: error: Argument 1 to "dump" of "YAML" has incompatible type "CommentedMap[Any, Any]"; expected "Path | SupportsWrite[str | bytes]"  [arg-type]
+ mitmproxy/tools/console/keymap.py:239: error: Item "StreamMark" of "Mark | StreamMark | None" has no attribute "get_snippet"  [union-attr]
+ mitmproxy/tools/console/keymap.py:239: error: Item "None" of "Mark | StreamMark | None" has no attribute "get_snippet"  [union-attr]
+ mitmproxy/tools/console/keymap.py:242: error: Item "None" of "Mark | StreamMark | None" has no attribute "line"  [union-attr]

paasta (https://github.com/yelp/paasta)
+ paasta_tools/cli/cmds/validate.py:44: error: Module "ruamel.yaml" has no attribute "SafeConstructor"  [attr-defined]
+ paasta_tools/cli/cmds/validate.py:339: error: "Type[BaseConstructor]" has no attribute "flatten_mapping"  [attr-defined]

spack (https://github.com/spack/spack)
+ lib/spack/spack/util/spack_yaml.py:201: error: Cannot infer type argument 1 of "add_representer" of "BaseRepresenter"  [misc]
+ lib/spack/spack/util/spack_yaml.py:204: error: Cannot infer type argument 1 of "add_representer" of "BaseRepresenter"  [misc]

check-jsonschema (https://github.com/python-jsonschema/check-jsonschema)
+ src/check_jsonschema/parsers/yaml.py:8: error: Module has no attribute "YAMLError"  [attr-defined]
+ src/check_jsonschema/parsers/yaml.py:18: error: Name "ruamel.yaml.SafeConstructor" is not defined  [name-defined]
+ src/check_jsonschema/parsers/yaml.py:27: error: Constructor? has no attribute "yaml_constructors"  [attr-defined]
+ src/check_jsonschema/parsers/yaml.py:28: error: Constructor? has no attribute "yaml_constructors"  [attr-defined]
+ src/check_jsonschema/parsers/yaml.py:59: error: Name "ruamel.yaml.YAMLError" is not defined  [name-defined]
+ src/check_jsonschema/parsers/yaml.py:66: error: Module has no attribute "YAMLError"  [attr-defined]
+ src/check_jsonschema/transforms/gitlab.py:22: error: Name "ruamel.yaml.BaseConstructor" is not defined  [name-defined]

github-actions[bot] avatar Sep 12 '24 19:09 github-actions[bot]

Diff from mypy_primer, showing the effect of this PR on open source code:

mitmproxy (https://github.com/mitmproxy/mitmproxy)
+ mitmproxy/optmanager.py:482: error: Need type annotation for "s"  [var-annotated]
+ mitmproxy/optmanager.py:496: error: Argument 1 to "dump" of "YAML" has incompatible type "CommentedMap[Any, Any]"; expected "Path | SupportsWrite[str | bytes]"  [arg-type]
+ mitmproxy/tools/console/keymap.py:239: error: Item "StreamMark" of "Mark | StreamMark | None" has no attribute "get_snippet"  [union-attr]
+ mitmproxy/tools/console/keymap.py:239: error: Item "None" of "Mark | StreamMark | None" has no attribute "get_snippet"  [union-attr]
+ mitmproxy/tools/console/keymap.py:242: error: Item "None" of "Mark | StreamMark | None" has no attribute "line"  [union-attr]

paasta (https://github.com/yelp/paasta)
+ paasta_tools/cli/cmds/validate.py:44: error: Module "ruamel.yaml" has no attribute "SafeConstructor"  [attr-defined]
+ paasta_tools/cli/cmds/validate.py:339: error: "Type[BaseConstructor]" has no attribute "flatten_mapping"  [attr-defined]

spack (https://github.com/spack/spack)
+ lib/spack/spack/util/spack_yaml.py:201: error: Cannot infer type argument 1 of "add_representer" of "BaseRepresenter"  [misc]
+ lib/spack/spack/util/spack_yaml.py:204: error: Cannot infer type argument 1 of "add_representer" of "BaseRepresenter"  [misc]

check-jsonschema (https://github.com/python-jsonschema/check-jsonschema)
+ src/check_jsonschema/parsers/yaml.py:8: error: Module has no attribute "YAMLError"  [attr-defined]
+ src/check_jsonschema/parsers/yaml.py:18: error: Name "ruamel.yaml.SafeConstructor" is not defined  [name-defined]
+ src/check_jsonschema/parsers/yaml.py:27: error: Constructor? has no attribute "yaml_constructors"  [attr-defined]
+ src/check_jsonschema/parsers/yaml.py:28: error: Constructor? has no attribute "yaml_constructors"  [attr-defined]
+ src/check_jsonschema/parsers/yaml.py:59: error: Name "ruamel.yaml.YAMLError" is not defined  [name-defined]
+ src/check_jsonschema/parsers/yaml.py:66: error: Module has no attribute "YAMLError"  [attr-defined]
+ src/check_jsonschema/transforms/gitlab.py:22: error: Name "ruamel.yaml.BaseConstructor" is not defined  [name-defined]

github-actions[bot] avatar Sep 12 '24 20:09 github-actions[bot]

Diff from mypy_primer, showing the effect of this PR on open source code:

mitmproxy (https://github.com/mitmproxy/mitmproxy)
+ mitmproxy/optmanager.py:482: error: Need type annotation for "s"  [var-annotated]
+ mitmproxy/tools/console/keymap.py:239: error: Item "StreamMark" of "Mark | StreamMark | None" has no attribute "get_snippet"  [union-attr]
+ mitmproxy/tools/console/keymap.py:239: error: Item "None" of "Mark | StreamMark | None" has no attribute "get_snippet"  [union-attr]
+ mitmproxy/tools/console/keymap.py:242: error: Item "None" of "Mark | StreamMark | None" has no attribute "line"  [union-attr]

paasta (https://github.com/yelp/paasta)
+ paasta_tools/cli/cmds/validate.py:339: error: "Type[BaseConstructor]" has no attribute "flatten_mapping"  [attr-defined]

spack (https://github.com/spack/spack)
+ lib/spack/spack/util/spack_yaml.py:201: error: Cannot infer type argument 1 of "add_representer" of "BaseRepresenter"  [misc]
+ lib/spack/spack/util/spack_yaml.py:204: error: Cannot infer type argument 1 of "add_representer" of "BaseRepresenter"  [misc]

check-jsonschema (https://github.com/python-jsonschema/check-jsonschema)
+ src/check_jsonschema/parsers/yaml.py:8: error: Module has no attribute "YAMLError"  [attr-defined]
+ src/check_jsonschema/parsers/yaml.py:59: error: Name "ruamel.yaml.YAMLError" is not defined  [name-defined]
+ src/check_jsonschema/parsers/yaml.py:66: error: Module has no attribute "YAMLError"  [attr-defined]

github-actions[bot] avatar Sep 15 '24 20:09 github-actions[bot]

Diff from mypy_primer, showing the effect of this PR on open source code:

mitmproxy (https://github.com/mitmproxy/mitmproxy)
+ mitmproxy/optmanager.py:482: error: Need type annotation for "s"  [var-annotated]
+ mitmproxy/tools/console/keymap.py:239: error: Item "StreamMark" of "Mark | StreamMark | None" has no attribute "get_snippet"  [union-attr]
+ mitmproxy/tools/console/keymap.py:239: error: Item "None" of "Mark | StreamMark | None" has no attribute "get_snippet"  [union-attr]
+ mitmproxy/tools/console/keymap.py:242: error: Item "None" of "Mark | StreamMark | None" has no attribute "line"  [union-attr]

paasta (https://github.com/yelp/paasta)
+ paasta_tools/cli/cmds/validate.py:339: error: "Type[BaseConstructor]" has no attribute "flatten_mapping"  [attr-defined]

github-actions[bot] avatar Sep 15 '24 21:09 github-actions[bot]

Diff from mypy_primer, showing the effect of this PR on open source code:

mitmproxy (https://github.com/mitmproxy/mitmproxy)
+ mitmproxy/optmanager.py:482: error: Need type annotation for "s"  [var-annotated]
+ mitmproxy/tools/console/keymap.py:239: error: Item "StreamMark" of "Mark | StreamMark | None" has no attribute "get_snippet"  [union-attr]
+ mitmproxy/tools/console/keymap.py:239: error: Item "None" of "Mark | StreamMark | None" has no attribute "get_snippet"  [union-attr]
+ mitmproxy/tools/console/keymap.py:242: error: Item "None" of "Mark | StreamMark | None" has no attribute "line"  [union-attr]

paasta (https://github.com/yelp/paasta)
+ paasta_tools/cli/cmds/validate.py:339: error: "Type[BaseConstructor]" has no attribute "flatten_mapping"  [attr-defined]

github-actions[bot] avatar Sep 15 '24 21:09 github-actions[bot]

Diff from mypy_primer, showing the effect of this PR on open source code:

mitmproxy (https://github.com/mitmproxy/mitmproxy)
+ mitmproxy/optmanager.py:482: error: Need type annotation for "s"  [var-annotated]
+ mitmproxy/tools/console/keymap.py:239: error: Item "StreamMark" of "Mark | StreamMark | None" has no attribute "get_snippet"  [union-attr]
+ mitmproxy/tools/console/keymap.py:239: error: Item "None" of "Mark | StreamMark | None" has no attribute "get_snippet"  [union-attr]
+ mitmproxy/tools/console/keymap.py:242: error: Item "None" of "Mark | StreamMark | None" has no attribute "line"  [union-attr]

paasta (https://github.com/yelp/paasta)
+ paasta_tools/cli/cmds/validate.py:339: error: "Type[BaseConstructor]" has no attribute "flatten_mapping"  [attr-defined]

github-actions[bot] avatar Sep 15 '24 21:09 github-actions[bot]

Currently stubtest fails to run:

error: not checking stubs due to mypy build errors:
/tmp/tmps3q1g0pn/lib/python3.11/site-packages/ruamel/yaml/main.py:47: error: Cannot assign to a type  [misc]
/tmp/tmps3q1g0pn/lib/python3.11/site-packages/ruamel/yaml/main.py:47: error: Incompatible types in assignment (expression has type "None", variable has type "type[CParser]")  [assignment]
/tmp/tmps3q1g0pn/lib/python3.11/site-packages/ruamel/yaml/main.py:47: error: Incompatible types in assignment (expression has type "None", variable has type "type[CEmitter]")  [assignment]
/tmp/tmps3q1g0pn/lib/python3.11/site-packages/ruamel/yaml/main.py:223: error: Unexpected keyword argument "loader"  [call-arg]
stubs/ruamel.yaml/_ruamel_yaml.pyi:22: note: Called function defined here
/tmp/tmps3q1g0pn/lib/python3.11/site-packages/ruamel/yaml/main.py:223: error: Unexpected keyword argument "loader"  [call-arg]
stubs/ruamel.yaml/_ruamel_yaml.pyi:22: note: Called function defined here

For the latter one, self.Parser is type[Parser | CParser] and mypy doesn't seem to be willing to narrow the type to type[Parser].

Not sure how to fix those.

Stubsabot dry run also fails to run:

Marking ruamel.yaml as obsolete since '0.15.99'
    asyncio.run(main())
  File "/opt/hostedtoolcache/Python/3.12.5/x64/lib/python3.12/asyncio/runners.py", line 194, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.5/x64/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.5/x64/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/home/runner/work/typeshed/typeshed/scripts/stubsabot.py", line 835, in main
    await suggest_typeshed_obsolete(update, session, action_level=args.action_level)
  File "/home/runner/work/typeshed/typeshed/scripts/stubsabot.py", line 735, in suggest_typeshed_obsolete
    with open(obsolete.stub_path / "METADATA.toml", "rb") as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'stubs/ruamel.yaml/METADATA.toml'

Help needed!

stevenlele avatar Sep 15 '24 22:09 stevenlele

Diff from mypy_primer, showing the effect of this PR on open source code:

mitmproxy (https://github.com/mitmproxy/mitmproxy)
+ mitmproxy/optmanager.py:482: error: Need type annotation for "s"  [var-annotated]
+ mitmproxy/tools/console/keymap.py:239: error: Item "StreamMark" of "Mark | StreamMark | None" has no attribute "get_snippet"  [union-attr]
+ mitmproxy/tools/console/keymap.py:239: error: Item "None" of "Mark | StreamMark | None" has no attribute "get_snippet"  [union-attr]
+ mitmproxy/tools/console/keymap.py:242: error: Item "None" of "Mark | StreamMark | None" has no attribute "line"  [union-attr]

paasta (https://github.com/yelp/paasta)
+ paasta_tools/cli/cmds/validate.py:339: error: "Type[BaseConstructor]" has no attribute "flatten_mapping"  [attr-defined]

github-actions[bot] avatar Sep 15 '24 22:09 github-actions[bot] 

mitmproxy/tools/console/keymap.py:239: error: Item "StreamMark" of "Mark | StreamMark | None" has no attribute "get_snippet"  [union-attr]
mitmproxy/tools/console/keymap.py:239: error: Item "None" of "Mark | StreamMark | None" has no attribute "get_snippet"  [union-attr]
mitmproxy/tools/console/keymap.py:242: error: Item "None" of "Mark | StreamMark | None" has no attribute "line"  [union-attr]

Remarks on Mark | StreamMark | None:

PyYAML is using the Any trick in nodes to relax the explicit None check:

https://github.com/python/typeshed/blob/dbe4d32a2a7e9c92689cdf850c00153c59ac2286/stubs/PyYAML/yaml/nodes.pyi#L5-L13

Sadly, the same trick can't be fully effective here. As you see, ruamel.yaml added a FileMark type for IO streams, which has no get_snippet method, so replacing None with Any won't help with the error on line 239 anyway. The line attribute check can be relaxed though. Let me know what you think.

https://github.com/python/typeshed/blob/1f10574d189b7b3ab56fc59c77a96741e217e852/stubs/ruamel.yaml/ruamel/yaml/error.pyi#L22-L37

stevenlele avatar Sep 15 '24 22:09 stevenlele

Diff from mypy_primer, showing the effect of this PR on open source code:

mitmproxy (https://github.com/mitmproxy/mitmproxy)
+ mitmproxy/optmanager.py:482: error: Need type annotation for "s"  [var-annotated]
+ mitmproxy/tools/console/keymap.py:239: error: Item "StreamMark" of "Mark | StreamMark | None" has no attribute "get_snippet"  [union-attr]
+ mitmproxy/tools/console/keymap.py:239: error: Item "None" of "Mark | StreamMark | None" has no attribute "get_snippet"  [union-attr]
+ mitmproxy/tools/console/keymap.py:242: error: Item "None" of "Mark | StreamMark | None" has no attribute "line"  [union-attr]

paasta (https://github.com/yelp/paasta)
+ paasta_tools/cli/cmds/validate.py:339: error: "Type[BaseConstructor]" has no attribute "flatten_mapping"  [attr-defined]

github-actions[bot] avatar Sep 16 '24 18:09 github-actions[bot]

Diff from mypy_primer, showing the effect of this PR on open source code:

mitmproxy (https://github.com/mitmproxy/mitmproxy)
+ mitmproxy/optmanager.py:482: error: Need type annotation for "s"  [var-annotated]
+ mitmproxy/tools/console/keymap.py:239: error: Item "StreamMark" of "Mark | StreamMark | None" has no attribute "get_snippet"  [union-attr]
+ mitmproxy/tools/console/keymap.py:239: error: Item "None" of "Mark | StreamMark | None" has no attribute "get_snippet"  [union-attr]
+ mitmproxy/tools/console/keymap.py:242: error: Item "None" of "Mark | StreamMark | None" has no attribute "line"  [union-attr]

paasta (https://github.com/yelp/paasta)
+ paasta_tools/cli/cmds/validate.py:339: error: "Type[BaseConstructor]" has no attribute "flatten_mapping"  [attr-defined]

github-actions[bot] avatar Sep 16 '24 23:09 github-actions[bot]