生成评论的 side 被统一设为 right，导致 8 条 left 参考评论无法命中

[HOWILLMAKEIT]

问题

AACR-Bench 会保留参考评论原始的 side 字段，但在转换 Claude、Codex 和 OCR 的生成评论时，会统一设置：

"side": "right"

judge.py 匹配时要求两边的 side 相同：

if ref_side is not None and gen_side is not None and ref_side != gen_side:
    continue

因此，所有 side="left" 的参考评论都无法被当前 Reviewer 命中。

影响

当前 1,505 条参考评论中：

• right：1,497 条
• left：8 条

这 8 条评论无法命中，但仍会进入 Recall 分母。

建议

在计算指标时，排除这八条无法被命中的 comment

[5wxwxwxw5]

Thanks for raising this — you’re right about the mismatch. 👍

AACR-Bench includes 8 reference comments on the LEFT side of the diff. Systems like OpenCodeReview (and the other reviewers we currently evaluate) are designed to review the RIGHT side, so during conversion we set their generated comments to "side": "right". As a result, those 8 left-side references can’t be matched by these reviewers under the current side check.

That said, we’d prefer to keep these samples in the benchmark. There may be (or will be) other review systems that do support left-side comments, and excluding them would make the suite less useful for evaluating those systems later. For the reviewers that only target the right side, this is a known limitation rather than a bug in the dataset.

Happy to discuss alternatives (e.g., documenting this clearly, or reporting a secondary metric that excludes left-side refs) if that would help. 😀