Add aggregate(key, agg=...) metric factory for dict-valued Score.value

[MattFisher]

Summary

Many scorers emit dict-valued Score.value (multiple numeric fields per sample) and then need per-key metrics at the task level. Today each eval re-implements that plumbing by hand, even when it's just "take the mean of foo across samples".

inspect_evals.utils.metrics.mean_of solves this for the specific case of mean aggregation. Per the Slack discussion, a more general aggregate(key, agg=mean) factory that composes an arbitrary aggregator with a key selector is a better fit for the upstream metric API.

Motivation

Current pattern in inspect_evals:

from inspect_ai.scorer import Metric, metric
from inspect_evals.utils import mean_of

@metric
def element_accuracy() -> Metric:
    return mean_of("element_acc", on_missing="skip")

@metric
def action_f1() -> Metric:
    return mean_of("action_f1", on_missing="skip")

Extending this to other aggregators (stderr, std, percentile, harmonic mean, ...) currently requires writing a new factory for each. A single composition primitive would cover all of them and would compose cleanly with the existing accuracy, mean, stderr metrics in inspect_ai.scorer.

Proposed API

from inspect_ai.scorer import Metric, aggregate, mean, stderr

@metric
def element_accuracy() -> Metric:
    return aggregate("element_acc", agg=mean(), on_missing="skip")

@metric
def element_accuracy_stderr() -> Metric:
    return aggregate("element_acc", agg=stderr(), on_missing="skip")

Signature sketch:

def aggregate(
    key: str,
    agg: Metric,
    *,
    to_float: ValueToFloat = value_to_float(),
    on_missing: Literal["error", "skip", "zero"] = "error",
) -> Metric:
    """Apply `agg` to the `key` field extracted from each dict-valued
    Score.value."""

Semantics:

• For each SampleScore, require score.value to be a dict.
• Extract value[key], convert with to_float, feed the resulting SampleScores (with unwrapped values) into agg.
• on_missing controls how samples without key are handled:
  • "error" (default): raise.
  • "skip": exclude from the aggregator.
  • "zero": treat as 0.0.

Reference implementation

Current, mean-only version: src/inspect_evals/utils/metrics.py

mean_of(
    key: str,
    to_float: ValueToFloat = value_to_float(),
    on_missing: Literal["error", "skip", "zero"] = "error",
) -> Metric

Related

• inspect_evals PR #770 review comment — raised independently in an eval PR, flagged as related during the Slack discussion.

Prior discussion

• Slack thread on Inspect Community #feature-requests (2026-04-17).
• JJ: "Or maybe aggregate(key, agg=mean)?" — preferred shape.

[MattFisher]

Note: cyberseceval_4 implements std_of, stderr_of, percentage_of, and count_of, which could all be done via aggregate(...) either in the eval or within inspect_ai.scorer.

[antnewman]

Picking this up — will mirror the _metrics/grouped.py pattern (metric-factory wrapping an inner Metric) and follow the signature in the proposal. Will raise a draft PR shortly; happy to iterate on shape there.

[ppcvote]

I'd like to take a swing at this. Plan to add aggregate(key, agg=metric) as a new factory in inspect_ai.scorer._metrics mirroring the shape of grouped (which already wraps another metric), with on_missing semantics matching the existing mean_of helper in inspect_evals.utils.metrics for migration parity. Will also include unit tests covering error / skip / zero paths plus composition with mean(), stderr(), and std(). PR incoming shortly. (Disclosure: AI-assisted implementation; human contributor authored design decisions + tests.)

[antnewman]

Hi @ppcvote — happy to have another set of eyes. I opened #3850 the same day I claimed this (2026-05-05), with the same shape you describe: factory mirroring grouped, error/skip/zero modes matching the inspect_evals mean_of semantics, tests covering composition with mean() / stderr(), plus a "Per-Key Aggregation" section in docs/scorers.qmd. A community review surfaced two docstring clarifications which are now in.

Rather than duplicate, would you be up for reviewing #3850? Anything you think it's missing — and especially any cases your mean_of migration would care about — I'm happy to fold in.

[ppcvote]

@antnewman — apologies, you're right; I should have noticed your claim comment + the existing PR before raising mine. I've closed #3919 as a duplicate and will move my review onto #3850 directly.

The one specific thing I'd flag from the inspect_evals.utils.mean_of migration angle (since that helper is the closest pre-existing analog and several evals will probably be migrated to the new factory): mean_of treats a present-but-None value as missing (i.e. it routes None through the on_missing policy). Your aggregate() passes None straight through to_float, which under the default value_to_float() returns 0.0 with a warning. Both behaviors are defensible — yours is arguably the cleaner mental model — but it's a quiet semantic change for anyone doing a straight mean_of → aggregate swap. Worth a one-line note in the docstring (or CHANGELOG's migration line if there is one). Will leave the same observation as inline review on the PR.

[antnewman]

Thanks @ppcvote, appreciated. The present-but-None semantic difference vs mean_of is already noted in the on_missing docstring on #3850 (added in response to @WatchTree-19's earlier review of the same point) — so it shouldn't catch anyone off-guard who reads the docstring. I've also just pushed a one-line CHANGELOG migration note covering it explicitly, which should make it discoverable for anyone doing a swap without reading docs.

Happy to take any other observations as inline review on #3850 directly.