PR reviewer and repo monitor skills ignore AUTOMATION_MODEL — automations run on the active profile, not their configured one

[VascoSch92]

Summary

github-pr-reviewer and github-repo-monitor build their conversation payload from the agent server's current settings instead of the automation's configured LLM profile. Neither script reads AUTOMATION_MODEL:

$ grep -c AUTOMATION_MODEL skills/github-pr-reviewer/scripts/main.py skills/github-repo-monitor/scripts/main.py
0
0

The result: an automation created from these skills ignores the profile selected for it and runs on whatever profile happens to be active in the UI at the moment it fires.

Where

skills/github-pr-reviewer/scripts/main.py:301-308

def _get_agent_dict(agent_url: str, api_key: str) -> dict:
    data = _fetch_settings(agent_url, api_key)
    llm = data.get("agent_settings", {}).get("llm", {})
    return {
        "kind": "Agent",
        "llm": llm,
        ...

skills/github-repo-monitor/scripts/main.py:439-452 has the same _get_agent_dict().

agent_settings.llm from GET /api/settings is the active profile, not the automation's.

This contradicts the documented contract

skills/openhands-automation/references/custom-automation.md:277-293 states that the service injects AUTOMATION_MODEL with the selected profile name and that scripts must honor it:

Calling workspace.get_llm() with no profile_name always uses the user's default LLM. The built-in prompt and plugin presets already follow the pattern above; custom scripts should too so the selected profile is honored regardless of trigger type or execution backend.

The service side works as documented — openhands/automation/dispatcher.py:253-254 sets env_vars["AUTOMATION_MODEL"], and the built-in presets (presets/prompt/sdk_main.py:95,290) consume it correctly. Only these two skill templates skip the step.

Observed on our shared OSS automation instance

agent-canvas 1.6.1, openhands-automation 1.3.1, local mode.

A triage automation generated from this pattern has profile gpt-5.6-sol configured. Every conversation it created today ran on a different model:

04:13:58Z  openai/gpt-5.5   usage_id=default
04:19:58Z  openai/gpt-5.5
04:36:00Z  openai/gpt-5.5
04:39:00Z  openai/gpt-5.5
05:37:04Z  openai/gpt-5.5

openai/gpt-5.5 was the active profile in the UI during that window. The inverse case confirms the mechanism: a sibling automation with no profile configured ran on openai/gpt-5.6-sol the previous evening — again the then-active profile. Runs from a preset-based automation on the same host, over the same dispatcher, honored their profile correctly:

AUTOMATION_MODEL: gpt-5.6-sol
=== GET_LLM ===
  profile: gpt-5.6-sol
  model: openai/gpt-5.6-sol

So the environment variable is delivered to these runs; the scripts just never read it.

Impact

• The per-automation LLM profile selector is silently a no-op for anything built from these two skills.
• Because the LLM is read live at fire time, anyone switching the active profile in the UI silently changes which model every such automation uses — on a shared instance, without touching any automation.
• PR review quality and cost drift with no signal in the run record. This blocks the outcome requested in Allow deterministic LLM profile selection for reviewer and QA automations automation#222.

Suggested fix

Resolve the profile when AUTOMATION_MODEL is set, and fall back to current behavior when it is not:

def _get_agent_dict(agent_url: str, api_key: str) -> dict:
    data = _fetch_settings(agent_url, api_key)
    llm = data.get("agent_settings", {}).get("llm", {})
    profile = os.environ.get("AUTOMATION_MODEL")
    if profile:
        req = urllib.request.Request(
            f"{agent_url}/api/profiles/{quote(profile, safe='')}",
            headers={"X-Session-API-Key": api_key, "X-Expose-Secrets": "plaintext"},
        )
        try:
            with urllib.request.urlopen(req, timeout=HTTP_TIMEOUT) as r:
                llm = json.loads(r.read())["config"]
            llm["usage_id"] = f"profile:{profile}"
        except urllib.error.HTTPError as exc:
            if exc.code != 404:
                raise
            # profile renamed or deleted after the automation was created
            print(f"profile {profile!r} not found; falling back to active profile")
    return {"kind": "Agent", "llm": llm, ...}

Two implementation notes:

• X-Expose-Secrets: plaintext is required — without it config.api_key comes back null. This mirrors what the SDK does in openhands/sdk/workspace/remote/base.py:364-380.
• StartConversationRequest exposes title_llm_profile and agent_profile_id but no field for the agent's LLM profile, so passing the resolved config inline is the only route over the REST API.

Worth auditing the other skills that create conversations directly for the same pattern.