crash with generator in list comprehension

[Akuli]

Crash Report

possibly related to #10188

Traceback

mypybug.py:7: error: INTERNAL ERROR -- Please try using mypy master on Github:
https://mypy.rtfd.io/en/latest/common_issues.html#using-a-development-mypy-build
Please report a bug at https://github.com/python/mypy/issues
version: 0.812
Traceback (most recent call last):
  File "mypy/semanal.py", line 4835, in accept
  File "mypy/nodes.py", line 1030, in accept
  File "mypy/semanal.py", line 3306, in visit_expression_stmt
  File "mypy/nodes.py", line 1956, in accept
  File "mypy/semanal.py", line 3883, in visit_list_comprehension
  File "mypy/nodes.py", line 1943, in accept
  File "mypy/semanal.py", line 3899, in visit_generator_expr
  File "mypy/nodes.py", line 1566, in accept
  File "mypy/semanal.py", line 3568, in visit_yield_from_expr
IndexError: list index out of range
mypybug.py:7: : note: use --pdb to drop into pdb

To Reproduce

from typing import Generator

def f() -> Generator[int, None, str]:
    yield 1
    return "lol"

[(yield from f()) for lel in range(5)]

Your Environment

• Mypy version used: 0.812
• Mypy command-line flags: none
• Mypy configuration options from mypy.ini (and other config files): none
• Python version used: 3.7.3

[JelleZijlstra]

This is a syntax error in 3.8 and later, presumably because it has no useful behavior: https://docs.python.org/3.10/whatsnew/3.8.html#changes-in-python-behavior. Perhaps mypy should just produce an error for it in any Python version.

[Bootstrap-Paradox]

Hey! I'm relatively new to Open Source Contribution, Is there anything I could be helpful for!😃

[alexbouayad]

This is a syntax error in 3.8 and later, presumably because it has no useful behavior: https://docs.python.org/3.10/whatsnew/3.8.html#changes-in-python-behavior. Perhaps mypy should just produce an error for it in any Python version.

It is indeed reported as a syntax error in 3.8 and later when compiling, but not when only building the AST, preventing mypy from catching the error before processing the graph.

The following raises SyntaxError (with python 3.9.9).

source = """from typing import Generator

def f() -> Generator[int, None, str]:
    yield 1
    return "lol"

[(yield from f()) for lel in range(5)]
"""

compile(source, filename="<string>", mode="exec")

The following raises no exception.

from ast import parse

parse(source)

[JelleZijlstra]

Good point! We should definitely catch this in mypy then. This should be a relatively easy thing to do for a new contributor.

[alexbouayad]

I can have a try!

Out of curiosity, should we regard the latter (that is, ast.parse does not raise SyntaxError) as a bug in python?

[JelleZijlstra]

I can have a try!

Great, thanks!

Out of curiosity, should we regard the latter (that is, ast.parse does not raise SyntaxError) as a bug in python?

I don't think so; there's a bunch of other things that are caught only after the ast is constructed and ast.parse doesn't catch them. For example, ast.parse("nonlocal x") works, but obviously you can't have a nonlocal in module scope.

Fortunately mypy doesn't crash on that one:

% mypy -c 'nonlocal x'
<string>:1: error: nonlocal declaration not allowed at module level
Found 1 error in 1 file (checked 1 source file)

[alexbouayad]

Ok, thanks!

[alexbouayad]

I just opened the PR #12048. My fix is quite concise. Maybe too much? (I may have missed something.) Also, please let me know if my PR doesn't meet the repo's standards.