Plugin development: Use of `get_dynamic_class_hook` causes deserialization of types to fail from `.mypy_cache`

[jackson-at-arista]

Hello folks. I'm working on a plugin that uses get_dynamic_class_hook to construct type constructors (TypeInfo) and class definitions (ClassDef) that don't exist in a module. I add these definitions to a MypyFile. Finally, I add this MypyFile to the modules dictionary provided by the SemanticAnalyzerPluginInterface API to convince mypy of my "fake" module.

This works wonderfully until a second run of my plugin when the .mypy_cache has been constructed. Before semantic analysis, mypy seems to be trying to read my fake module, because in the cache names in real modules have a target that refers to the fake module. For instance,

{
  ".class": "MypyFile",
  "_fullname": "RealModule",
   ...
  "names": {
    ".class": "SymbolTable",
    ...
    "SomeType": {
      ".class": "SymbolTableNode",
      "kind": "Gdef",
      "node": {
        ".class": "TypeAlias",
        ...
        "fullname": "RealModule.SomeType",
        ...
        "target": "FakeModule.SomeType"
      }
    },
    ...

The stacktrace is below. mypy fails to resolve FakeModule.SomeType because FakeModule doesn't exist in its modules dictionary. I don't think my use of get_dynamic_class_hook is unusual, so I think there's a hole in my understanding of this hook.

...
-> process_graph(graph, manager)
  /usr/lib/python3.9/site-packages/mypy/build.py(3333)process_graph()
-> process_fresh_modules(graph, prev_scc, manager)
  /usr/lib/python3.9/site-packages/mypy/build.py(3414)process_fresh_modules()
-> graph[id].fix_cross_refs()
  /usr/lib/python3.9/site-packages/mypy/build.py(2110)fix_cross_refs()
-> fixup_module(self.tree, self.manager.modules, self.options.use_fine_grained_cache)
  /usr/lib/python3.9/site-packages/mypy/fixup.py(52)fixup_module()
-> node_fixer.visit_symbol_table(tree.names, tree.fullname)
  /usr/lib/python3.9/site-packages/mypy/fixup.py(156)visit_symbol_table()
-> self.visit_type_info(value.node)
  /usr/lib/python3.9/site-packages/mypy/fixup.py(115)visit_type_info()
-> self.current_info = save_info
  /usr/lib/python3.9/site-packages/mypy/fixup.py(158)visit_symbol_table()
-> value.node.accept(self)
  /usr/lib/python3.9/site-packages/mypy/nodes.py(1037)accept()
-> return visitor.visit_var(self)
  /usr/lib/python3.9/site-packages/mypy/fixup.py(211)visit_var()
-> v.type.accept(self.type_fixer)
  /usr/lib/python3.9/site-packages/mypy/types.py(1444)accept()
-> return visitor.visit_instance(self)
  /usr/lib/python3.9/site-packages/mypy/fixup.py(230)visit_instance()
-> inst.type = lookup_fully_qualified_typeinfo(
  /usr/lib/python3.9/site-packages/mypy/fixup.py(367)lookup_fully_qualified_typeinfo()
-> stnode = lookup_fully_qualified(name, modules, raise_on_missing=not allow_missing)
  /usr/lib/python3.9/site-packages/mypy/lookup.py(31)lookup_fully_qualified()
-> assert "." in head, f"Cannot find module for {name}"

[ljluestc]

from typing import Optional, Callable, Any, Dict
from mypy.plugin import Plugin, SemanticAnalyzerPluginInterface, DynamicClassDefContext
from mypy.nodes import MypyFile, SymbolTable, SymbolTableNode, TypeInfo, ClassDef
from mypy.types import Instance

class MyDynamicClassPlugin(Plugin):
    def get_dynamic_class_hook(self, fullname: str) -> Optional[Callable[[DynamicClassDefContext], None]]:
        if fullname == "real.module.name":
            return my_dynamic_class_hook
        return None

def my_dynamic_class_hook(ctx: DynamicClassDefContext) -> None:
    fake_module_name = "fake.module"
    fake_class_name = "FakeClass"

    # Create ClassDef and TypeInfo for the fake class
    class_def = ClassDef(fake_class_name, [], ctx.api.cur_mod_id)
    class_def.fullname = f"{fake_module_name}.{fake_class_name}"
    
    info = TypeInfo(SymbolTable(), class_def, ctx.api.cur_mod_id)
    info.mro = [info]  # Minimal MRO for a single class
    info.names["__init__"] = SymbolTableNode(kind=MDEF, node=None)
    class_def.info = info

    # Add the fake class to the current module's symbol table
    ctx.api.add_symbol_table_node(fake_class_name, SymbolTableNode(GDEF, info))

    # Create and register the fake module
    if fake_module_name not in ctx.api.modules:
        fake_mypyfile = MypyFile([], [], ctx.api.cur_mod_id, False, False, False, None)
        ctx.api.modules[fake_module_name] = fake_mypyfile
    
    ctx.api.modules[fake_module_name].names[fake_class_name] = SymbolTableNode(GDEF, info)

def plugin(version: str) -> Plugin:
    return MyDynamicClassPlugin

# Ensure the plugin is registered in mypy.ini
# [mypy]
# plugins = path.to.plugin_file

[jackson-at-arista]

Hi ljluestc, thanks for your help! Your excerpt is similar to the approach I'm currently taking, except that we may be running different mypy versions because your MypyFile constructor looks different than mine. I'm on mypy 1.10.0.

I'm not sure your excerpt solves my issue though, because looking at my stacktrace control is never transferred to the plugin before mypy crashes. It doesn't look like mypy reaches its semantic analysis stage and my_dynamic_class_hook (in your case) never fires.