Calculating custom aggregate metrics for evals

[olinguyen]

Question

Hello!

What's the recommendation for implementing custom aggregate metrics like precision/recall for evals?
There's an existing ReportCaseAggregate but that seems specific for calculating the average of scores.

There's a few workarounds on top of my head:

1. Implement my own EvaluationReport and overwrite the dataset's evaluate() function to call an aggregate report
2. Write my own custom script to calculate metrics from the list of cases (ignoring any Evaluator)

Ideally, it seems we should be able to have an Evaluator compute function that runs on a list of predictions and labels, similar to sklearn's precision_score(y_true, y_pred) or huggingface's evaluate's compute(predictions, references)

Additional Context

No response

[algorithmica-repository]

Adding to the above question and #1431, we need support from evaluator to return list/tuple of numbers instead of a scalar. Here is my usecase.

class BertEvaluator(Evaluator):
    def evaluate(self, ctx: EvaluatorContext[str, str]) -> float:
        p, r, f = score([ctx.output], [ctx.expected_output], lang="en")
        return f.item()


class RougeEvaluator(Evaluator):
    def evaluate(self, ctx: EvaluatorContext[str, str]) -> float:
        scores = Rouge().get_scores(ctx.output, ctx.expected_output)
        rouge_1 = scores[0]["rouge-1"]
        return rouge_1["f"]

In BertEvaluator, I want to return p.item(), r.item() and f.item() as a list. Same is the case for RougeEvalutor.

[github-actions]

This issue is stale, and will be closed in 3 days if no reply is received.

[dmontagu]

So I think the body of this issue from @olinguyen and the comment from @algorithmica-repository are representing two distinct issues.

---

I'll first reply to @algorithmica-repository's comment.

I think from an implementation perspective it should be straightforward to support returning a list of outputs. However, I'm not sure how those should be treated by an application displaying the results, and I would prefer if there was an obvious interpretation.

For the existing types (boolean, string, number) it's straightforward — booleans are treated as assertions, strings as labels, numbers as numeric scores.

Can you describe the use case for returning a list of values? Note that evaluators can already return a dictionary of label-to-score, so you could do return {'p': p.item(), 'r': r.item(), 'f': f.item()} (or with more detailed labels) and then you'd even get labels for which is which, and be able to get averages of each, etc. But is there some advantage to using a list instead of a dict for this?

Either way, if you want to discuss this more, please create a new issue and just link to this comment, since it is on a different topic than the original post.

---

Regarding @olinguyen's original question, I understand the request and think it's reasonable.

Right now, we have the include_averages argument to pydantic_evals.reporting.EvaluationReport.console_table and similar, and under the hood that boolean flag determines whether the ReportCaseAggregate.average method is called.

Perhaps a better way to generalize this would be to add an argument to the EvaluationReport.console_table and similar methods roughly along the lines of aggregations: Sequence[Callable[[list[ReportCase]], ReportCaseAggregate]] = (pydantic_evals.reporting.ReportCaseAggregate.average,). This would provide you a place to pass your own custom aggregation functions.

Would that be roughly along the lines you were looking for @olinguyen? PR for that would be welcome, also any feedback would be useful in case the ReportCaseAggregate is too rigid in some way, or you otherwise take issue with that approach to passing custom aggregation functions.