Logfire for LLM Evals inside pytest decorator do not work

[warp10-simonezambonim]

Initial Checks

• I confirm that I'm using the latest version of Pydantic AI
• I confirm that I searched for my issue in https://github.com/pydantic/pydantic-ai/issues before opening this issue

Description

I am using pytest in my project and I wanted to include the LLM evals in the same way, however the logging is not sent to logfire when running pytest.

Example Code

import logfire
import pytest
from pydantic_evals import Dataset
from agents import my_agent
from models import InputModel, OutputModel

TEST_FILE = "llm-evals/file.yaml"

...

@pytest.mark.asyncio
async def test_therapeutic_area_agent():
    dataset = Dataset[InputModel, OutputModel].from_file(TEST_FILE)
    with logfire.span("test"):
        logfire.info(f"Loaded dataset with {len(dataset.cases)} cases")

        report = await dataset.evaluate(my_agent, name="my_agent")
        report.print(include_input=True, include_output=True, include_expected_output=True, include_durations=True)
        logfire.info(f"Average score {report.averages().assertions:.2f}")

Python, Pydantic AI & LLM client version

pydantic-ai-slim = {extras = ["anthropic", "openai"], version = "^0.0.55"}
pydantic-evals = {extras = ["logfire"], version = "^0.0.55"}
AnthropicProvider

[dmontagu]

So, first question — did you call logfire.configure somewhere that gets executed as part of the test run? If not, then that might explain why it's not getting sent anywhere — the configure call (with an appropriate token, etc.) has to happen for it to send data anywhere. You can do this in a pytest fixture, and you can mark that fixture with autouse=True if you want to apply it to every test without needing to be manually included in the test function signature.

If you did call logfire.configure and are still hitting this, let us know.

[warp10-simonezambonim]

I am sorry, I might have missed to include that on my sample code

from shared.observability_handler import configure_observability

@pytest.fixture(autouse=True, scope="session")
def setup_observability():
    configure_observability()

and shared/observability_handler.py

import logfire
from pydantic_ai import Agent

from shared.settings import shared_settings

def configure_observability():
    logfire.configure(
        environment=shared_settings.logfire_environment,
        token=shared_settings.logfire_token,
        service_name="pharma-agent",
    )
    Agent.instrument_all()
    logfire.instrument_httpx(capture_all=True)

If I call it outside from pytest it is working.

Thank you for looking into it!

[peppermint-ai-lab]

Can confirm. This does not work for us either. Here is the repro project.

repro-pydantic.zip