Issue 1 — execute_command tool description omits the PascalCase naming convention
- Problem: The tool signature injected into the agent's system prompt only says
args (dict): Command arguments, with no mention that the underlying cluster methods (e.g. _move_to_level) require PascalCase kwargs like Level and TransitionTime. When the agent sends snake_case, Python raises TypeError, which gets wrapped as a generic 400 Bad Request. The agent never sees the real cause and cannot self-correct.
- Evidence chain: the docstring in
tools.py L216–228, the implementation in level_control.py L134–147, a programmatic repro, and the full trajectory of qt2_feasible_seed_69 where the agent hit MoveToLevel three times and got 400 each time.
- Scope: at least 5 QT2-Feasible episodes (~10%) fail purely from this naming mismatch — seeds 5, 40, 53, 69, 74. Same pitfall applies to
SetpointRaiseLower, MoveWithOnOff, StepWithOnOff.
- Proposed fix: (A) extend the docstring with PascalCase examples; (B) normalize
args keys at the API boundary so the interface is case-insensitive.
Issue 2 — QT1 evaluation penalizes agents that report accurate real-time PM10
- Problem:
eval.goals[].current_value snapshots PM10 at T=0, but the simulator keeps ticking during agent execution and air purifiers actively reduce PM10. The agent faithfully queries and reports the current value; because it no longer matches the stale snapshot, the LLM judge scores B across the board.
- Evidence chain: the full
qt1_feasible_seed_4 case (snapshot 20.0 vs runtime 0.0 → B) plus a five-episode drift table (seeds 4/40/20/6/33).
- Root cause: unlike QT2 — which explicitly calls
reset_simulation() + fast_forward_to() for a deterministic baseline — QT1 evaluation never rewinds to T=0. The judge implicitly cross-checks tool observation against current_value in the goal.
- Proposed fix: (A) at benchmark generation, warm up N ticks before recording
current_value; (B) apply a ±15% tolerance window; (C) restrict the judge to answer-vs-observation faithfulness and treat current_value as a reference hint, not strict ground truth.
Issue 1 —
execute_commandtool description omits the PascalCase naming conventionargs (dict): Command arguments, with no mention that the underlying cluster methods (e.g._move_to_level) require PascalCase kwargs likeLevelandTransitionTime. When the agent sends snake_case, Python raisesTypeError, which gets wrapped as a generic 400 Bad Request. The agent never sees the real cause and cannot self-correct.tools.pyL216–228, the implementation inlevel_control.pyL134–147, a programmatic repro, and the full trajectory ofqt2_feasible_seed_69where the agent hitMoveToLevelthree times and got 400 each time.SetpointRaiseLower,MoveWithOnOff,StepWithOnOff.argskeys at the API boundary so the interface is case-insensitive.Issue 2 — QT1 evaluation penalizes agents that report accurate real-time PM10
eval.goals[].current_valuesnapshots PM10 at T=0, but the simulator keeps ticking during agent execution and air purifiers actively reduce PM10. The agent faithfully queries and reports the current value; because it no longer matches the stale snapshot, the LLM judge scores B across the board.qt1_feasible_seed_4case (snapshot 20.0 vs runtime 0.0 → B) plus a five-episode drift table (seeds 4/40/20/6/33).reset_simulation()+fast_forward_to()for a deterministic baseline — QT1 evaluation never rewinds to T=0. The judge implicitly cross-checks tool observation againstcurrent_valuein the goal.current_value; (B) apply a ±15% tolerance window; (C) restrict the judge to answer-vs-observation faithfulness and treatcurrent_valueas a reference hint, not strict ground truth.