Skip to content

Best-of-N: run the same agent instance multiple times for variance sampling #45

Description

@DavidWells

Idea

Allow running the exact same agent instance (same provider + model + effort) multiple times on one task, to sample variance and pick the best output — "best of N."

Deferred as a non-goal in the multi-instance agent configuration plan (docs/plans/multi-instance-agent-configuration.md), where instances are unique by provider:model:effort. This issue captures the follow-on.

Sketch

  • A count (or repeated label) on an instance entry fans out to N identical runs:
    agents:
      - { agent: claude, model: claude-opus-5, effort: high, count: 3 }
  • Instance ids disambiguated by an index/label (claude:claude-opus-5:high#1..3).
  • Pairs naturally with the arena / side-by-side comparison view (netlify-react-ui PR #25842) and a judge/synthesis step to rank the N outputs.

Why later

The base program keys runs by instance id and rejects exact duplicates; adding count/labels + a ranking affordance is additive on top of that once the instance model lands.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions