Crash when parsing function with unexpected await

[jamescooke]

Current behaviour

Yapf crashes when attempting to format a function which contains an unexpected await.

cat > my_bad.py

def run() -> None:
    await do_thing()
^d

Install latest yapf from PyPI into a clean Python 3.12 venv and run it against my_bad.py:

pip install yapf==0.40.2
yapf my_bad.py

An IndexError is raised. Here's the full traceback:

Traceback (most recent call last):
  File "/tmp/tmp.8dsxCLfnr9/venv312/lib/python3.12/site-packages/yapf/yapflib/yapf_api.py", line 198, in FormatCode
    tree = pytree_utils.ParseCodeToTree(unformatted_source)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/tmp.8dsxCLfnr9/venv312/lib/python3.12/site-packages/yapf/pytree/pytree_utils.py", line 113, in ParseCodeToTree
    tree = parser_driver.parse_string(code, debug=False)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/tmp.8dsxCLfnr9/venv312/lib/python3.12/site-packages/yapf_third_party/_ylib2to3/pgen2/driver.py", line 188, in parse_string
    return self.parse_tokens(tokens, debug)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/tmp.8dsxCLfnr9/venv312/lib/python3.12/site-packages/yapf_third_party/_ylib2to3/pgen2/driver.py", line 157, in parse_tokens
    if p.addtoken(type, value, (prefix, start)):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/tmp.8dsxCLfnr9/venv312/lib/python3.12/site-packages/yapf_third_party/_ylib2to3/pgen2/parse.py", line 230, in addtoken
    return self._addtoken(ilabel, type, value, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/tmp.8dsxCLfnr9/venv312/lib/python3.12/site-packages/yapf_third_party/_ylib2to3/pgen2/parse.py", line 313, in _addtoken
    raise ParseError('bad input', type, value, context)
yapf_third_party._ylib2to3.pgen2.parse.ParseError: bad input: type=1, value='do_thing', context=(' ', (2, 10))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/tmp.8dsxCLfnr9/venv312/lib/python3.12/site-packages/yapf/__init__.py", line 241, in _FormatFile
    reformatted_code, encoding, has_change = yapf_api.FormatFile(
                                             ^^^^^^^^^^^^^^^^^^^^
  File "/tmp/tmp.8dsxCLfnr9/venv312/lib/python3.12/site-packages/yapf/yapflib/yapf_api.py", line 88, in FormatFile
    reformatted_source, changed = FormatCode(
                                  ^^^^^^^^^^^
  File "/tmp/tmp.8dsxCLfnr9/venv312/lib/python3.12/site-packages/yapf/yapflib/yapf_api.py", line 201, in FormatCode
    raise errors.YapfError(errors.FormatErrorMsg(e))
                           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/tmp.8dsxCLfnr9/venv312/lib/python3.12/site-packages/yapf/yapflib/errors.py", line 37, in FormatErrorMsg
    return '{}:{}:{}: {}'.format(e.args[1][0], e.args[1][1], e.args[1][2], e.msg)
                                 ~~~~~~^^^
IndexError: tuple index out of range

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/tmp.8dsxCLfnr9/venv312/bin/yapf", line 8, in <module>
    sys.exit(run_main())
             ^^^^^^^^^^
  File "/tmp/tmp.8dsxCLfnr9/venv312/lib/python3.12/site-packages/yapf/__init__.py", line 380, in run_main
    sys.exit(main(sys.argv))
             ^^^^^^^^^^^^^^
  File "/tmp/tmp.8dsxCLfnr9/venv312/lib/python3.12/site-packages/yapf/__init__.py", line 139, in main
    changed = FormatFiles(
              ^^^^^^^^^^^^
  File "/tmp/tmp.8dsxCLfnr9/venv312/lib/python3.12/site-packages/yapf/__init__.py", line 217, in FormatFiles
    changed |= _FormatFile(filename, lines, style_config, no_local_style,
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/tmp.8dsxCLfnr9/venv312/lib/python3.12/site-packages/yapf/__init__.py", line 251, in _FormatFile
    raise errors.YapfError(errors.FormatErrorMsg(e))
                           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/tmp.8dsxCLfnr9/venv312/lib/python3.12/site-packages/yapf/yapflib/errors.py", line 37, in FormatErrorMsg
    return '{}:{}:{}: {}'.format(e.args[1][0], e.args[1][1], e.args[1][2], e.msg)
                                 ~~~~~~^^^
IndexError: tuple index out of range

Error is reproducible with Pythons 3.11 and 3.12. Also with latest version from git installed with pip install -U git+https://github.com/google/yapf/ which gives the version:

yapf @ git+https://github.com/google/yapf/@6a92535967543aa979461fbe2da63a9b6f7dd595

Expected behaviour

We would expect Yapf to do a nice job of warning of the bad syntax, rather than raising an exception.

For example, if we run with Python:

python my_bad.py

  File "/tmp/tmp.8dsxCLfnr9/my_bad.py", line 2
    await do_thing()
    ^^^^^^^^^^^^^^^^
SyntaxError: 'await' outside async function

And if we write "bad" Python, then Yapf usually warns us:

cat > other.py

def thing()() -> None:
    pass
^d

yapf other.py

yapf: other.py:1:12: expected ':'