Metadata in Eval description in Dataset

[loukdeloijer-x]

Description

When loading an eval dataset from a yaml file, it would be nice to also provide metadata to a specific eval if you are performing for example multiple LLMJudge evals.

The PDF data I'm trying to evaluate contains codes with texts which falls in different categories. The textsnippets are lengthy, so a Contains won't do it.

example:

101010:

• Header: textsnippet A
• Subtext: textsnippet B

cases:

• name: 101010
  inputs:
  prompt: "What is the data under code 101010
  evaluators:

  • LLMJudge:
    rubric: "Answer should mention textsnippet A"
    model: azure:gpt-4o
    metadata:
    type: header

  • LLMJudge:
    rubric: "Answer should mention tetsnippet B"
    model: azure:gpt-4o
    metadata:
    type: subtext

References

No response

[dmontagu]

Do you want the LLMJudge to make use of this metadata during its evaluation?

---

If you do want the LLMJudge evaluation to be aware of the metadata, I think it would make more sense to do one of the following:

• Update the rubric to just explicitly mention the metadata
• Create a new evaluator (possibly by subclassing LLMJudge, possibly just a whole new class) that has the relevant "type" metadata as a field, and incorporates it into the prompt used by the LLMJudge.

If just manually incorporating the metadata into the prompt for each case is undesirable for you but you would be potentially willing to implement your own evaluator, I'm happy to try to demonstrate how that would work. In particular, here are some possibly-useful snippets:

@dataclass
class MyLLMJudge(Evaluator[object, object, object]):
    """Judge whether the output of a language model meets the criteria of a provided rubric.

    If you do not specify a model, it uses the default model for judging. This starts as 'openai:gpt-4o', but can be
    overridden by calling [`set_default_judge_model`][pydantic_evals.evaluators.llm_as_a_judge.set_default_judge_model].
    """

    rubric: str
    metadata_type: Literal['header', 'subtext']
    model: models.Model | models.KnownModelName | None = None
    include_input: bool = False

    async def evaluate(
        self,
        ctx: EvaluatorContext[object, object, object],
    ) -> EvaluationReason:
        rubric = f'Judge the {self.metadata_type} of the output. {self.rubric}'
        if self.include_input:
            from .llm_as_a_judge import judge_input_output

            grading_output = await judge_input_output(ctx.inputs, ctx.output, rubric, self.model)
        else:
            from .llm_as_a_judge import judge_output

            grading_output = await judge_output(ctx.output, rubric, self.model)
        return EvaluationReason(value=grading_output.pass_, reason=grading_output.reason)

    def build_serialization_arguments(self):
        result = super().build_serialization_arguments()
        # always serialize the model as a string when present; use its name if it's a KnownModelName
        if (model := result.get('model')) and isinstance(model, models.Model):
            result['model'] = f'{model.system}:{model.model_name}'

        # Note: this may lead to confusion if you try to serialize-then-deserialize with a custom model.
        # I expect that is rare enough to be worth not solving yet, but common enough that we probably will want to
        # solve it eventually. I'm imagining some kind of model registry, but don't want to work out the details yet.
        return result

Note: this is just a copy/paste of the LLMJudge implementation, but if this is along the lines of what you are looking for (and maybe even if it's not), I'll rework LLMJudge to have a public build_rubric method which by default just returns self.rubric but can be overridden and make use of fields added in a subclass. So then your custom evaluator definition could just look like this:

@dataclass
class MyLLMJudge(LLMJudge):
    metadata_type: str = field(kw_only=True)
    def build_rubric(self):
        return f'Judge the {self.metadata_type} of the output. {self.rubric}'

(You would then just need to include MyLLMJudge in the provided sequence of custom evaluators when serializing/deserializing. Happy to help with this if you have questions about how to make it work.)

---

If you don't want the LLMJudge to be aware of the metadata, then it makes sense to not want to need to subclass LLMJudge, but then it's less clear to me why this is a beneficial feature. (Perhaps in this case you could explain more.)