`validate_structured_result` returns empty while usage shows there's an output (also it doesn't go to except)

[seunggs]

I keep getting empty dict {} as validated_restructured_result response. Here's the code:

class DocumentModelResponse(TypedDict, total=False):
    model_response: str
    document_md: str

...

# Run agent with streaming
async with agent.run_stream(
    user_prompt=prompt, deps=agent_deps, message_history=message_history
) as result:
    # Stream the model's response with structured validation
    async for message, last in result.stream_structured(debounce_by=0.01):
        try:
            validated_response = await result.validate_structured_result(
                message,
                allow_partial=not last,
            )

            yield f"event: document_messages\ndata: {json.dumps(validated_response)}\n\n"
        except ValidationError as e:
            raise ModelRetry("some error message")

    # After streaming is complete, get usage stats
    usage = result.usage()
    logfire.info(
        "Received LLM response",
        **{
            "request_tokens": usage.request_tokens,
            "response_tokens": usage.response_tokens,
            "total_tokens": usage.total_tokens,
        },
    )
...

This is strange since usage log shows reponse_tokens which indicates that the model actually sent back a response:

{
  'request_tokens': 1076, 'response_tokens': 1039, 'total_tokens': 2115
}

It also doesn't fall back to except with ValidationError when empty dict is returned.

Any guidance on this would be much appreciated. Thanks!

[dmontagu]

Can you print out messsage in your loop? I.e., add the following line:

        try:
            print(message)  # <---------- this line
            validated_response = await result.validate_structured_result(
                message,
                allow_partial=not last,
            )

if the produced output is e.g. using different keys than DocumentModelResponse it might explain the issue, or otherwise there's probably some insight to be had in the data that you are receiving before validation.

[seunggs]

@dmontagu This is what I see:

ModelResponse(
    parts=[
        ToolCallPart(
            tool_name="final_result", args={}, tool_call_id="toolu_01KDJh1975Ce3GSf3yhqqSob", part_kind="tool-call"
        )
    ],
    model_name="claude-3-5-sonnet-latest",
    timestamp=datetime.datetime(2025, 2, 13, 0, 11, 1, 789946, tzinfo=datetime.timezone.utc),
    kind="response",
)

Usage log shows:

{'request_tokens': 2440, 'response_tokens': 1038, 'total_tokens': 3478}

[seunggs]

Turns out it doesn't just happen with result. stream_structured - it happens sporadically with result.stream as well:

# Run agent with streaming
async with agent.run_stream(
    user_prompt=prompt, deps=agent_deps, message_history=message_history
) as result:
    # Stream the model's response with structured validation
    async for message in result.stream():
        print(message)
        yield f"event: document_messages\ndata: {json.dumps(message)}\n\n"

    # After streaming is complete, get usage stats
    usage = result.usage()
    logfire.info(
        "Received LLM response",
        **{
            "request_tokens": usage.request_tokens,
            "response_tokens": usage.response_tokens,
            "total_tokens": usage.total_tokens,
        },
    )