dmypy "Daemon crashed!" with KeyError: 'sqlalchemy.orm'

[rwarren]

Below is a repeatable dmypy invokation sequence showing a crash due to KeyError. Is there a way to get better debug output to help resolve this?

$ dmypy run -- --config-file /home/russ/pcode/edms/server/../env/mypy.ini --strict /home/russ/pcode/edms/server/deleteme.py 
Daemon started
deleteme.py:3:3: error: Unsupported operand types for + ("int" and "str")  [operator]
Found 1 error in 1 file (checked 21 source files)
$
$ dmypy run -- --config-file /home/russ/pcode/edms/server/../env/mypy.ini --strict /home/russ/pcode/edms/server/deleteme.py 
deleteme.py:3:3: error: Unsupported operand types for + ("int" and "str")  [operator]
Found 1 error in 1 file (checked 1 source file)
$
$ dmypy run -- --config-file /home/russ/pcode/edms/server/../env/mypy.ini --strict /home/russ/pcode/edms/server/deleteme.py 
deleteme.py:3:3: error: Unsupported operand types for + ("int" and "str")  [operator]
Found 1 error in 1 file (checked 1 source file)
$
$ dmypy run -- --config-file /home/russ/pcode/edms/server/../env/mypy.ini --strict /home/russ/pcode/edms/shared/deployment_check/deployment_check.py 
Success: no issues found in 1 source file
$
$ dmypy run -- --config-file /home/russ/pcode/edms/server/../env/mypy.ini --strict /home/russ/pcode/edms/server/server/EventLogger.py 
Daemon crashed!
Traceback (most recent call last):
  File "mypy/dmypy_server.py", line 223, in serve
  File "mypy/dmypy_server.py", line 266, in run_command
  File "mypy/dmypy_server.py", line 325, in cmd_run
  File "mypy/dmypy_server.py", line 387, in check
  File "mypy/dmypy_server.py", line 561, in fine_grained_increment_follow_imports
  File "mypy/dmypy_server.py", line 686, in direct_imports
KeyError: 'sqlalchemy.orm'

[rwarren]

The file that causes the daemon crash has this in it:

import sqlalchemy.orm as orm

[indigoviolet]

I see this quite often as well, with other keys (eg. pathlib, fastapi etc) -- I have not been able to discern a pattern:

Suspicious state from syntax checker python-dmypy: Flycheck checker python-dmypy returned 2, but its output contained no errors: Daemon crashed!
Traceback (most recent call last):
  File "mypy/dmypy_server.py", line 223, in serve
  File "mypy/dmypy_server.py", line 266, in run_command
  File "mypy/dmypy_server.py", line 325, in cmd_run
  File "mypy/dmypy_server.py", line 387, in check
  File "mypy/dmypy_server.py", line 561, in fine_grained_increment_follow_imports
  File "mypy/dmypy_server.py", line 686, in direct_imports
KeyError: 'pathlib'

[gvanrossum]

@rwarren To reproduce this we would need to have the source code of your application and the contents of your mypy.ini file, and the mypy and Python versions used. Unfortunately this bug is not easy to debug without a reliable repro, so having this information would be much appreciated.

[gvanrossum]

Ironically/embarrassingly, I got this exact traceback by running mypy over itself in an odd sequence. This is repeatable on my Mac:

• Create and activate new virtual enviroment using Python 3.8 and install mypy
• Change into the mypy repo root (make sure there's no .mypy_cache directory)
• Run dmypy run -- --config-file mypy_self_check.ini mypy; this reported 14 errors related to pytest
• Pip-install requirements from test-requirements.txt and mypy-requirements.txt
• Re-run the same dmypy command; this reported 116 errors about various things
• Re-run the same dmypy command; this gave the above crash

Daemon crashed!
Traceback (most recent call last):
  File "mypy/dmypy_server.py", line 223, in serve
    # If we are crashing, report the crash to the client
  File "mypy/dmypy_server.py", line 266, in run_command
    # Command functions (run in the server via RPC).
  File "mypy/dmypy_server.py", line 325, in cmd_run
    def cmd_check(self, files: Sequence[str],
  File "mypy/dmypy_server.py", line 387, in check
    self.fscache.flush()
  File "mypy/dmypy_server.py", line 561, in fine_grained_increment_follow_imports
    # infinite looping if there are any self edges. (Self
  File "mypy/dmypy_server.py", line 686, in direct_imports
    """Return the direct imports of module not included in seen."""
KeyError: '_pytest.assertion.rewrite'

My conclusion is that (in my case) the still-running mypy daemon process is confused by the pip-install that happened while it was running. But this is not exactly what @rwarren experienced.

One way to debug this would be to put a breakpoint() call in direct_imports(), something like this:

diff --git a/mypy/dmypy_server.py b/mypy/dmypy_server.py
index 157850b39..20fb39b96 100644
--- a/mypy/dmypy_server.py
+++ b/mypy/dmypy_server.py
@@ -684,6 +684,7 @@ class Server:
                        module: Tuple[str, str],
                        graph: mypy.build.Graph) -> List[BuildSource]:
         """Return the direct imports of module not included in seen."""
+        if module[0] not in graph: breakpoint()
         state = graph[module[0]]
         return [BuildSource(graph[dep].path, dep)
                 for dep in state.dependencies]

and then run the daemon server in the foreground (dmypy daemon). Probably need to add the --config-file flag there too. You then open a second terminal window and run the above dmypy run commands there. Use pdb to poke around in the daemon state when the breakpoint hits.

(UPDATE: Because I run this in the mypy repo root it's running mypy from source there. That's essential, otherwise you get the mypyc-optimized production version which isn't debuggable by mortals.)

[gvanrossum]

I'm looking into this. That function (direct_imports()) doesn't get called on any of the previous runs -- and in the final run I only see three calls, all related to _pytest.

[gvanrossum]

So my patch was wrong, since the error actually occurs a few lines further down, in the list comprehension.

I found a simpler repro, using a file that just contains import pytest, and the following recipe:

• (UPDATE:) Modify mypy/dmypy_server.py to catch KeyError and call breakpoint() when it happens, to root around in the debugger.

• In a separate window, run python -m mypy.dmypy daemon -- --config-file mypy_self_check.ini (this is where breakpoint() will enter pdb)

• Run the following script:

pip uninstall -y pytest
python -m mypy.dmypy check dd.py  # Error about pytest not found
pip install pytest
python -m mypy.dmypy check dd.py  # Errors about pytest internals
python -m mypy.dmypy check dd.py  # Crash; dep == '_pytest.assertion.rewrite'

However, debugging the crash on the last run is not all that useful because what goes wrong is clearly the previous run -- it shouldn't complain about pytest internals. I think what's going on is that somehow the failure in the first run leaves behind a dummy entry for pytest which is not completely erased when the second run finds pytest present (due to the pip install pytest).

The first error from (from the second run) is this:

/Users/guido/vv/lib/python3.8/site-packages/pytest/__init__.py:5: error: Module 'pytest' has no attribute 'collect' 
[attr-defined]
    from . import collect
    ^

This is the very first import in pytest/__init__.py. However, other imports in that file are apparently still followed (probably due to the pre-scan which is used to construct the import graph?), and we get lots of other errors about modules in _pytest. (The pytest distribution structure is weird. It installs two packages, pytest and _pytest, with the latter being the actual implementation, and the former just being a shim providing the public interface. Both have a py.typed marker.) This is the first _pytest error:

/Users/guido/vv/lib/python3.8/site-packages/_pytest/capture.py:151: error: Function is missing a return type annotation
 [no-untyped-def]
    def pytest_load_initial_conftests(early_config: Config):
    ^

That seems legit (it's due to the flag disallow_incomplete_defs = True in mypy_self_check.ini) but such errors ought to be suppressed for code in site-packages`, and in fact when we restart the daemon we get zero errors.

At this point I'm pretty stumped, so I could use some help (though I may have some more time over the weeked to look into this -- I'm stuck inside indefinitely with the ash rain).

[gvanrossum]

@JukkaL The offending code was introduced by your PR #8559 ("Work towards following imports in mypy daemon"). Do you remember being unsure about any parts of it?

[gvanrossum]

Perhaps the most useful debugging technique here is to set DEBUG_FINE_GRAINED = True in mypy/build.py. In particular the second run ended with skipping a whole bunch of modules -- here are the first and last few error messages:

update once: pytest in 0.626s - 76 left
skip 'pytest.collect' (module not in import graph)
skip 'attr' (module not in import graph)
skip 'attr.filters' (module not in import graph)
skip 'attr.converters' (module not in import graph)
skip 'attr.setters' (module not in import graph)
skip 'attr.validators' (module not in import graph)
skip '_pytest.reports' (module not in import graph)
skip '_pytest.runner' (module not in import graph)
skip '_pytest.terminal' (module not in import graph)
skip '_pytest.warnings' (module not in import graph)
skip '_pytest.mark.expression' (module not in import graph)
skip 'doctest' (module not in import graph)
skip 'unittest' (module not in import graph)
skip 'unittest.async_case' (module not in import graph)
.
.
.
skip 'email.contentmanager' (module not in import graph)
skip 'importlib.metadata' (module not in import graph)
skip '_pytest.assertion.rewrite' (module not in import graph)
skip '_pytest.assertion.truncate' (module not in import graph)
skip '_pytest.assertion.util' (module not in import graph)
skip '_pytest._code.source' (module not in import graph)
process target with error: _pytest.debugging
process target with error: _pytest.nodes.Node._repr_failure_py
process target with error: _pytest.fixtures.fixture
process target with error: _pytest.config.PytestPluginManager.__init__
process target with error: _pytest.mark.structures
process target with error: _pytest._code.code
process target with error: _pytest.cacheprovider
process target with error: _pytest.nodes
process target with error: _pytest.cacheprovider.Cache.for_config
process target with error: _pytest.mark.structures.MarkGenerator.__getattr__
process target with error: _pytest.fixtures.FixtureManager.getfixtureinfo
process target with error: _pytest.config
process target with error: _pytest.config.Config.__init__
process target with error: _pytest.deprecated
process target with error: _pytest.capture
process target with error: dd
process target with error: _pytest.mark.KeywordMatcher.from_item
process target with error: pytest
process target with error: _pytest.assertion
process target with error: _pytest.fixtures.pytest_sessionstart
process target with error: _pytest.config.get_config
process target with error: _pytest.fixtures
triggered: []

(UPDATE: The first skip line, skip 'pytest.collect' (module not in import graph) seems the most important to debug next, since that module exists in the pytype package -- so something in the code that determines the import graph must not be looking right for modules in py.typed-marked packages.)

[gvanrossum]

@JukkaL Is it normal that update_module() in FineGrainedBuildManager is called many times for the same module? I put a print in there and it prints "pytest" many times (over 100x before I gave up) on the 2nd run in my repro script (the run right after pytest was installed, which produces many errors starting with the one about pytest.collect).

[gvanrossum]

I think I've reduced it to an import cycle between a module and the package containing it, if the import cycle is far enough away from the main program. (site-packages is irrelevant.)

==> ./main.py <==
import pytest

==> ./pytest/__init__.py <==
from _pytest import assertion

==> ./attr/__init__.py <==
Attribute = 1
from . import filters

==> ./attr/filters.py <==
from . import Attribute

==> ./_pytest/config/__init__.py <==
import attr

==> ./_pytest/__init__.py <==

==> ./_pytest/assertion/__init__.py <==
import _pytest.config

To be clear, the recipe still involves:

• pip uninstall -y pytest
• Remove all of the above directories (pytest, _pytest, attrs) but keep main.py
• dmypy check main.py # Error about pytest not found
• restore all three directories
• dmypy check main.py # Clean!
• dmypy check main.py # Crash

At the point of the crash, these modules (plus the usual builtins) are present in the graph:

• _pytest
• _pytest.assertion
• pytest
• main

But these are missing:

• _pytest.config
• attr
• attr.filters

If I take a shortcut in main.py e.g. import _pytest.assertion the crash disappears. If I remove the import cycle the crash disappears.

[gvanrossum]

Slowly getting to the crux of the matter. I tried a simpler set of files:

==> main.py <==
import aa

==> aa.py <==
import bb

==> bb.py <==
import cc

==> cc.py <==
A = 1
import dd

==> dd.py <==
from cc import A
print(A+"")  # E: Unsupported operand types for + ("int" and "str")

This doesn't crash, but from the output logged by DEBUG_FINE_GRAINED = True it seems that it doesn't go deep enough into the import graph -- the error doesn't get detected on the first few runs.

• The first run after creating aa.py-dd.py, the daemon updates 'aa' and skips 'bb', 'cc' and 'dd'. No errors from dmypy.
• The second run the daemon updates 'bb' and skips 'cc' and 'dd'. No errors from dmypy.
• The third run the daemon 'cc' and skips just 'dd'. No errors from dmypy.
• The fourth run the daemon finally updates 'dd' and doesn't skip anything. Here dmypy finally reports the error.
• Runs after that it doesn't update or skip anything, and dmypy reports the same errors.

[gvanrossum]

The chain doesn't have to be as long. Here's a diff with a test that passes, but should fail on the command labeled #1; the import cycle is between aa and bb:

diff --git a/test-data/unit/daemon.test b/test-data/unit/daemon.test
index d7dad66b5..db24c90a3 100644
--- a/test-data/unit/daemon.test
+++ b/test-data/unit/daemon.test
@@ -295,3 +295,27 @@ from foo import foo
 def bar() -> None:
     x = foo('abc')  # type: str
     foo(arg='xyz')
+
+[case testDaemonTwoPartDissonance]
+$ dmypy start --log-file log.txt -- --no-error-summary
+Daemon started
+$ dmypy check main.py
+main.py:1: error: Cannot find implementation or library stub for module named 'aa'
+main.py:1: note: See https://mypy.readthedocs.io/en/latest/running_mypy.html#missing-imports
+== Return code: 1
+$ {python} -c "import shutil; shutil.copy('aa.py.2', 'aa.py'); shutil.copy('bb.py.2', 'bb.py')"
+$ dmypy check main.py  #1
+$ dmypy check main.py  #2
+bb.py:2: error: Unsupported operand types for + ("int" and "str")
+== Return code: 1
+$ dmypy check main.py  #3
+bb.py:2: error: Unsupported operand types for + ("int" and "str")
+== Return code: 1
+[file main.py]
+import aa
+[file aa.py.2]
+A = 1
+import bb
+[file bb.py.2]
+from  aa import A
+A + ""  # E: Unsupported operand types for + ("int" and "str")