[Contribution] Feature: Persistent system prompt for multi-turn datasets

[hatemfaheem]

Problem

When benchmarking multi-turn conversations with a custom multi_turn dataset, there is no first-class way to apply a single system prompt that persists across every turn of a session.

Authors currently have two unsatisfying options:

1. Repeat the system prompt in every turn — verbose, error-prone, and it changes the per-turn payload shape.
2. Add a leading {"role": "system", ...} turn — but today this is dispatched as its own standalone, user-less request. That:
   • emits an extra request with no user message (incomplete prompt in some cases)
   • inflates the turn count (a 1-system + 2-user session reports as 3 turns), and
   • skews per-turn metrics.

There is no way to say "this is the conversation's system prompt, prepend it to each turn" from a custom dataset file.

Proposal

Treat a leading, text-only {"role": "system", ...} turn in a multi_turn dataset as the conversation-level system prompt: hoist it into conversation.system_message (the field already consumed by the chat endpoint and --shared-system-prompt-length) instead of dispatching it as a turn.

Result: the prompt is prepended to every turn's request, persists across the whole session, and does not consume a turn slot.

{"session_id": "chat_1", "turns": [
  {"role": "system", "text": "You are a terse assistant. Answer in one sentence."},
  {"text": "What is machine learning?"},
  {"text": "Give me an example."}
]}

→ runs as 2 turns, with the system prompt included in each request.

Scope / requirements

• Hoist only a leading, text-only system turn. A mid-conversation system turn, or one carrying image/audio/video media, must remain a normal turn (no silent content drop).
• Do not change fixed-benchmark loaders. The multi_turn loader is the base class for the SpeedBench family; SPEED-Bench rows legitimately lead with a system message that is part of the benchmark definition. Those loaders must keep the system message as a dispatched turn so their turn counts and acceptance-rate metrics are unaffected.
• A system-only session must remain runnable. If the only turn is a leading system turn, hoisting would leave a turn-less conversation; the scheduler cannot dispatch that (it raises "turn index out of range" and the run hangs). This degenerate input must fail fast / behave as it did before the feature, not hang.
• Document the behavior in the custom-dataset tutorial.