Pydantic Eval: Evaluator does not support list of int

[josead]

Initial Checks

• I confirm that I'm using the latest version of Pydantic AI
• I confirm that I searched for my issue in https://github.com/pydantic/pydantic-ai/issues before opening this issue

Description

When returning {"a": [2]} from my evaluator method I receive error.

I traced back the issue and this is the line that fails

# run_evaluator
results = _EVALUATOR_OUTPUT_ADAPTER.validate_python(raw_results)

Example Code

Python, Pydantic AI & LLM client version

0.0.55

[dmontagu]

Can you speak more about why it's useful to return a list of integers/etc.?

One of the benefits of the current design is that in a UI for displaying the evaluation results, each of the currently-supported types has a clear semantic meaning — booleans correspond to assertions, numbers correspond to numeric scores, and strings correspond to labels. And there is a clear way to handle things like aggregation etc.

@alexmojaki has mentioned on numerous occasions wanting to have evaluators return arbitrary-shape data but I've generally pushed back that it wasn't clear to me why that would be useful and it would prevent you from doing automated aggregations, etc.

However, I am not opposed to supporting this in principle, especially if it is a common request. But understanding why it's useful might make it easier to justify the effort involved and downstream consequences. Can you share a bit more?