diff --git a/AI_INSTALL.md b/AI_INSTALL.md new file mode 100644 index 000000000..2e5d6338b --- /dev/null +++ b/AI_INSTALL.md @@ -0,0 +1,184 @@ +# AI_INSTALL — paste this to your AI assistant + +**For humans:** You are using an AI IDE (Trae / Cursor / Claude Code / Codex / Windsurf / Cline) +and it is already logged in. You do **not** need an API key. Copy this whole file to your +AI assistant and say: *"Install RepoBrain in this project by following AI_INSTALL.md."* +Then just answer any question it asks. When it finishes, ask it anything about your codebase. + +**面向用户:** 你正在用某个 AI IDE(Trae / Cursor / Claude Code / Codex / Windsurf / Cline), +而且已经登录。你**不需要 API key**。把这个文件整段贴给你的 AI 助手,对它说: +*"照着 AI_INSTALL.md 在这个项目里装好 RepoBrain。"* 然后回答它问你的问题即可。 +装完后,直接问它任何关于你代码库的问题。 + +--- + +## Instructions for the AI assistant / 给 AI 助手的指令 + +You are an installation assistant. Your goal: install and configure **RepoBrain** in the +user's **current project** so that afterwards the user can ask questions about their codebase +and you (the AI) will answer them grounded in real code. + +你是安装助手。目标:在用户**当前项目**里安装并配置 **RepoBrain**,让用户之后能问关于代码库的 +问题,而你(AI)能基于真实源码回答。 + +Follow the steps **in order**. If a step fails, diagnose and fix it before continuing. +Report progress to the user in their language. Never invent an API key or paste secrets. + +按顺序执行。某步失败就先排查修复再继续。用用户的语言汇报进度。**绝不**编造 API key 或粘贴密钥。 + +### Step 0 — Confirm you can run shell commands / 确认你能执行 shell + +This script only works if you (the AI) can execute terminal commands in the user's project. +If you cannot run commands, stop and tell the user to follow `INSTALL.md` manually instead. + +只有当你能在用户项目里执行终端命令时这套流程才成立。如果你不能执行命令,请停止并告诉用户 +改用 `INSTALL.md` 手动安装。 + +### Step 1 — Check Python / 检查 Python + +RepoBrain needs **Python 3.10+**. Run: + +```bash +python3 --version +``` + +If it is missing or below 3.10, install a newer Python (prefer the user's platform tool: +`brew install python@3.12` on macOS, `pyenv install`, or the system package manager) and make +sure `python3` on PATH is ≥ 3.10 before continuing. + +如果缺失或低于 3.10,装一个更新的 Python(优先用平台工具:macOS 用 `brew install python@3.12`, +或 `pyenv`、系统包管理器),确认 PATH 上的 `python3` ≥ 3.10 再继续。 + +### Step 2 — Install RepoBrain / 安装 RepoBrain + +Prefer `pipx` (isolated, does not pollute the environment). Fall back to `pip --user`. +Install the **engine** (`rb-ask`, `rb-refresh`, `rb-mcp`) and inject the **CLI** (`rb`, `rb init`): + +优先用 `pipx`(隔离、不污染环境),失败则回退 `pip --user`。安装**引擎**(`rb-ask` / `rb-refresh` / +`rb-mcp`)并注入 **CLI**(`rb` / `rb init`): + +```bash +# Preferred: pipx +pipx install "git+https://github.com/study8677/repobrain.git#subdirectory=engine" +pipx inject --force --include-apps repobrain-engine "git+https://github.com/study8677/repobrain.git#subdirectory=cli" + +# Fallback: pip --user +python3 -m pip install --user "git+https://github.com/study8677/repobrain.git#subdirectory=engine" +python3 -m pip install --user "git+https://github.com/study8677/repobrain.git#subdirectory=cli" +``` + +Verify: `rb-ask --help` prints usage. If the command is not found, the install bin directory +is not on PATH — add it (the installer prints the path) and re-check. + +验证:`rb-ask --help` 能打印用法。若提示找不到命令,是安装目录不在 PATH 上——把它加进 PATH +(安装器会打印路径)再验证。 + +### Step 3 — Configure a zero-key backend / 配置零-key 后端 + +**Prefer no API key.** Detect a logged-in local headless CLI and write a host-runner `.env`. +Check in this order and use the **first** one that is available and logged in: + +**优先零 API key。** 探测本机已登录的无头 CLI,写入 host-runner 的 `.env`。按下面顺序检查, +用**第一个**可用且已登录的: + +1. **Trae** — `command -v trae-cli` and `trae-cli login status`. If OK, write to `.env`: + + ```bash + RB_HOST_RUNNER=generic + RB_HOST_COMMAND=trae-cli exec --cd {workspace} --sandbox read-only --skip-git-repo-check --ephemeral -o {output_file} + RB_HOST_OUTPUT_MODE=file + RB_HOST_TIMEOUT_SECONDS=240 + ``` + +2. **Codex** — `command -v codex` and `codex login status`. If OK, write to `.env`: + + ```bash + RB_HOST_RUNNER=codex + RB_HOST_MODEL=gpt-5.3-codex-spark + RB_HOST_TIMEOUT_SECONDS=240 + RB_HOST_MAX_CONTEXT_CHARS=60000 + ``` + +3. **Claude Code** — `command -v claude` (logged-in state is implicit). If present, write to `.env`: + + ```bash + RB_HOST_RUNNER=generic + RB_HOST_COMMAND=claude -p --add-dir {workspace} + RB_HOST_OUTPUT_MODE=stdout + RB_HOST_TIMEOUT_SECONDS=240 + ``` + +4. **Fallback — no local CLI found.** Only if none of the above is available, tell the user a + zero-key backend needs a logged-in Trae/Codex/Claude, and offer to run `rb-setup` so they can + paste an API key instead. Do not fabricate a key. + + **回退——没探测到本地 CLI。** 只有在以上都不可用时,告诉用户零-key 需要一个已登录的 + Trae/Codex/Claude,并提议运行 `rb-setup` 让他贴 API key。不要编造 key。 + +Write `.env` to the **project root** and make sure it is git-ignored: + +把 `.env` 写到**项目根目录**,并确保它被 git 忽略: + +```bash +grep -qxF '.env' .gitignore 2>/dev/null || echo '.env' >> .gitignore +``` + +### Step 4 — Initialize the project / 初始化项目 + +Drop the RepoBrain convention files (`AGENTS.md`, `CLAUDE.md`, `.trae/rules/…`, `.cursorrules`, …) +so any AI IDE — including you — automatically knows to call `rb-ask`: + +放入 RepoBrain 约定文件(`AGENTS.md` / `CLAUDE.md` / `.trae/rules/…` / `.cursorrules` 等), +让任何 AI IDE(包括你自己)以后都自动调用 `rb-ask`: + +```bash +rb init . +``` + +### Step 5 — Smoke test / 冒烟自测 + +Ask RepoBrain one question. The first call auto-builds the knowledge base (no separate +`rb-refresh` needed), then answers. Show the answer to the user: + +问 RepoBrain 一个问题。首次调用会自动建库(无需单独 `rb-refresh`)再回答。把答案展示给用户: + +```bash +rb-ask "What does this project do? / 这个项目是做什么的?" --workspace . +``` + +For programmatic use, `--json` returns `{answer, sources, limitations, workspace, question}`. + +程序化调用可加 `--json`,返回 `{answer, sources, limitations, workspace, question}`。 + +### Step 6 — Report / 汇报 + +Tell the user, in their language: + +用用户的语言告诉他: + +> ✅ RepoBrain is installed and configured for **zero-API-key** mode (driving your logged-in +> ``). From now on, just ask me anything about this codebase — I will use RepoBrain to +> answer, grounded in real code with file paths and line numbers. The knowledge base refreshes +> itself automatically; run `rb-refresh --workspace .` only to force a full rebuild. +> +> ✅ RepoBrain 已装好并配置为**零 API key**模式(驱动你已登录的 ``)。以后直接问我 +> 关于这个代码库的任何问题即可——我会用 RepoBrain 基于真实源码作答,带文件路径和行号。 +> 知识库会自动刷新;只有强制全量重建时才需手动跑 `rb-refresh --workspace .`。 + +--- + +## Capability boundaries (be honest with the user) / 能力边界(如实告诉用户) + +Zero-key mode drives a **single-turn, tool-free** local CLI, so `rb-refresh` degrades gracefully: + +零-key 模式驱动的是**单轮、无工具调用**的本地 CLI,因此 `rb-refresh` 会优雅降级: + +- **module docs / map** — generated by your local CLI ✅ +- **conventions** — collapses the multi-hop handoff swarm into a single-turn agent ⚠️ +- **git insights** — deterministic pre-extracted git data, no LLM narration ⚠️ + +`rb-ask` Q&A works fully (single-turn). If the user later gets an API key, `rb-setup` switches +to the full-capability, tool-using refresh automatically — a configured API backend always wins. + +`rb-ask` 问答完全可用(单轮)。用户以后若拿到 API key,`rb-setup` 会自动切到全功能、带工具的 +refresh——配置了 API 后端时永远优先用它。 diff --git a/INSTALL.md b/INSTALL.md index ab39bc70c..b769be9e7 100644 --- a/INSTALL.md +++ b/INSTALL.md @@ -12,8 +12,8 @@ 1. **Marketplace add** — clones the plugin manifest into Claude Code's cache. 2. **Install** — first session triggers `hooks/install_engine.py`, which installs the engine (`rb-ask`, `rb-refresh`, `rb-mcp`) and injects the `rb` CLI into the same `pipx` environment. It falls back to `pip --user` or prints manual commands if installation fails. Cross-platform (macOS / Linux / Windows). -3. **Setup** — interactive: choose your LLM provider (OpenAI / DeepSeek / Groq / 阿里灵积 / NVIDIA / Ollama), paste your API key, writes a `.env` to the current project root and ensures it's git-ignored. For local Codex users, setup can instead write `RB_HOST_RUNNER=codex` for experimental no-API-key `rb-ask`. -4. **Refresh** — runs `rb-refresh` directly and builds `.repobrain/` for the current project. The first refresh creates the project knowledge directory automatically. Full LLM refresh requires an API key; Codex host-runner mode uses scan-only refresh artifacts. +3. **Setup** — interactive: it **first detects any logged-in local headless CLI** (Codex / Trae / Claude / Gemini) and offers a **no-API-key local host runner** as the easiest default — no key to paste, RepoBrain drives your existing CLI login for both `rb-ask` and `rb-refresh`. If you prefer a hosted model, it also offers API-key providers (OpenAI / DeepSeek / Groq / 阿里灵积 / NVIDIA / Ollama). Either way it writes a `.env` to the current project root and ensures it's git-ignored. +4. **Refresh** — runs `rb-refresh` directly and builds `.repobrain/` for the current project. The first refresh creates the project knowledge directory automatically. With an API key you get the full tool-using / handoff refresh; in local host-runner mode refresh still runs its tool-free stages (module docs, map) through your CLI and automatically degrades tool/handoff stages (conventions → single-turn agent, git insights → deterministic pre-extracted data). 5. **Ask** — runs `rb-ask` directly and queries the refreshed project knowledge base. MCP is optional. If you want tool-style integration in an MCP-compatible host, @@ -57,10 +57,13 @@ You can also keep using the raw CLI directly: `rb-refresh --workspace ` If your Codex build supports MCP and you want tool-style integration, register `rb-mcp --workspace ` separately in your Codex MCP configuration. -### Codex host-runner mode without an API key +### Local host-runner mode without an API key -If you are only using RepoBrain locally and your Codex CLI is already logged -in with ChatGPT, you can use the experimental host runner for `rb-ask`: +If you are only using RepoBrain locally and have a logged-in headless CLI, you can drive it as +the backend for **both `rb-ask` and `rb-refresh`** — no API key. The easiest path is `rb-setup`, +which detects your CLI and writes this for you; to configure it by hand, pick your runner: + +**Codex** (built-in preset): ``` codex login status @@ -69,16 +72,34 @@ RB_HOST_RUNNER=codex RB_HOST_MODEL=gpt-5.3-codex-spark RB_HOST_TIMEOUT_SECONDS=240 RB_HOST_MAX_CONTEXT_CHARS=60000 -RB_REFRESH_SCAN_ONLY=1 EOF +``` + +**Trae / any headless CLI** (generic runner via `RB_HOST_COMMAND`): -rb-refresh --workspace . # scan-only artifacts, no API key +``` +trae-cli login status +cat >> .env <<'EOF' +RB_HOST_RUNNER=generic +RB_HOST_COMMAND=trae-cli exec --cd {workspace} --sandbox read-only --skip-git-repo-check --ephemeral -o {output_file} +RB_HOST_OUTPUT_MODE=file +RB_HOST_TIMEOUT_SECONDS=240 +EOF +``` + +Then: + +``` +rb-refresh --workspace . # builds the knowledge base through your local CLI, no API key rb-ask "what does this project do?" --workspace . ``` -This mode is ask-only and depends on the user's local Codex installation and -login. It is not a hosted product backend and does not replace API-key-backed -full refresh. +This depends on the user's local CLI installation and login. It is not a hosted product backend. +Because a local CLI is **single-turn and tool-free**, refresh runs module docs and the map +through your CLI and degrades the tool/handoff stages: conventions collapses to a single-turn +agent, and git insights fall back to deterministic pre-extracted data. A configured API backend +always wins and keeps the full tool-using / handoff refresh. Prefer a pure scan (no LLM at all)? +Set `RB_REFRESH_SCAN_ONLY=1`. ## DeepSeek Harness @@ -130,7 +151,7 @@ Same four commands ship to both hosts. Claude Code namespaces them as `/repobrai | Claude Code | Codex CLI | What it does | |---|---|---| -| `/repobrain:rb-setup` | `/rb-setup` | **First-time setup** — interactive `.env` writer (LLM provider + key + model, or local Codex host runner) | +| `/repobrain:rb-setup` | `/rb-setup` | **First-time setup** — interactive `.env` writer (logged-in local CLI = no key, or an API-key provider + model) | | `/repobrain:rb-refresh [quick]` | `/rb-refresh [quick]` | Rebuild / incrementally update the project knowledge base | | `/repobrain:rb-ask ` | `/rb-ask ` | Routed Q&A on the current codebase | | `/repobrain:rb-init ` | `/rb-init ` | Scaffold a new multi-agent repo from this template | diff --git a/README.md b/README.md index 9a0394e15..665a68895 100644 --- a/README.md +++ b/README.md @@ -55,7 +55,7 @@ /plugin marketplace add study8677/repobrain /plugin install repobrain@repobrain -# 2 — Pick LLM provider, build the knowledge base +# 2 — Configure a backend (logged-in local CLI = no key, or paste an API key), build the knowledge base /repobrain:rb-setup /repobrain:rb-refresh @@ -67,6 +67,14 @@ > [Head-to-head benchmark ↓](#head-to-head-eval-repobrain-vs-codex-cli-vs-claude-code-2026-05-09) > Codex CLI users — drop the `repobrain:` prefix; the same four slash commands ship there too. +### 🧠 Already using an AI IDE? Let it install RepoBrain for you — no API key. + +If you use Trae / Cursor / Claude Code / Codex (and it's logged in), you don't need to touch +pip or an API key. **Paste [`AI_INSTALL.md`](AI_INSTALL.md) to your AI assistant and say +"install RepoBrain in this project by following it."** It detects your logged-in CLI, wires up +a zero-key backend, initializes the project, and self-tests — then you just ask it anything +about your codebase. + --- ## Why RepoBrain? @@ -159,7 +167,7 @@ Full report (data, methodology, per-cell tables, caveats): # Claude Code /plugin marketplace add study8677/repobrain /plugin install repobrain@repobrain -/repobrain:rb-setup # interactive: pick LLM provider, paste API key, writes .env +/repobrain:rb-setup # interactive: use a logged-in local CLI (Codex/Trae/Claude, no key) or paste an API key; writes .env /repobrain:rb-refresh # first refresh auto-creates .repobrain/ /repobrain:rb-ask "How does this project work?" @@ -237,13 +245,13 @@ If installation or provider setup looks wrong, run `rb doctor --workspace .`. ### `rb-setup` — first-time configuration -Run this **once per project**, right after installing the plugin. Interactive picker for the LLM provider (OpenAI / DeepSeek / Groq / 阿里灵积 / NVIDIA NIM / Ollama local / any OpenAI-compatible endpoint), then writes `.env` to the project root with `OPENAI_BASE_URL`, `OPENAI_API_KEY`, `OPENAI_MODEL`, `RB_ASK_TIMEOUT_SECONDS`. It can also write an explicit local Codex host-runner config for experimental no-API-key `rb-ask`. Also ensures `.env` is in `.gitignore`. Skip it if you already have a working `.env`. +Run this **once per project**, right after installing the plugin. Interactive picker that first **detects the headless CLIs you're already logged into** (Codex / Trae / Claude / Gemini) and offers a **no-API-key local host-runner** as the most convenient option — no key to paste, RepoBrain just drives your existing CLI login for both `rb-ask` and `rb-refresh`. If you'd rather use a hosted model, it also offers the API-key providers (OpenAI / DeepSeek / Groq / 阿里灵积 / NVIDIA NIM / Ollama local / any OpenAI-compatible endpoint). Either way it writes `.env` to the project root — `RB_HOST_RUNNER` + `RB_HOST_COMMAND` for a local CLI, or `OPENAI_BASE_URL` / `OPENAI_API_KEY` / `OPENAI_MODEL` for a provider — and ensures `.env` is in `.gitignore`. Skip it if you already have a working `.env`. ### `rb-refresh` — build / refresh the knowledge base Deploys the multi-agent cluster to read your code: each module gets its own Agent that produces a knowledge doc under `.repobrain/agents/*.md`, plus a `map.md` routing index. Run after install, after significant code changes, or when `rb-ask` returns stale answers. The first refresh auto-creates `.repobrain/` — no separate init step needed. Pass `quick` for an incremental update, `failed-only` to rerun only previously failed modules. -Time: a few minutes for small repos, longer for large ones. Requires `rb-setup` to have completed. Full LLM refresh requires an API-key/OpenAI-compatible provider; local host-runner mode can use `RB_REFRESH_SCAN_ONLY=1 rb-refresh --workspace .` for scan artifacts. +Time: a few minutes for small repos, longer for large ones. Requires `rb-setup` to have completed. Works with either backend: an API-key/OpenAI-compatible provider runs the full LLM refresh, while a **local host-runner** (Codex / Trae / Claude / …) runs the tool-free stages (module docs, `map.md`) through your logged-in CLI and automatically degrades the tool/handoff stages (conventions, git insights) to deterministic output — no API key needed. Add `RB_REFRESH_SCAN_ONLY=1` only if you want a fast structure-only index with no LLM narration at all. ### `rb-ask` — routed Q&A on the codebase @@ -251,6 +259,15 @@ The **main reason this plugin exists**. Routes your question to the right Module Requires a knowledge base — if you see "no index" or empty answers, run `rb-refresh` first. +**Calling from another AI / script?** Add `--json` for a stable, parseable envelope instead of human-formatted text — this is the lightweight way to let any agent that can run a shell command query RepoBrain, no MCP server required: + +```bash +rb-ask "How does auth work?" --workspace . --json +# → {"answer": "...", "sources": [...], "limitations": [...], "workspace": "...", "question": "..."} +``` + +On failure, `--json` keeps stdout empty and writes `{"error": "..."}` to stderr with a non-zero exit code, so a calling agent can branch cleanly. It runs the same engine as everything else, so it works with an API-key provider **or** a no-API-key local host runner. See [Let another AI call RepoBrain (CLI, no MCP)](#let-another-ai-call-repobrain-cli-no-mcp). + ### `rb-init` — scaffold a new multi-agent repo Creates a **new** project from the RepoBrain template. Two modes: `quick` (fast scaffold, clean copy) and `full` (adds runtime profile, `.env`, mission file, sandbox config, optional `git init`). This is for **starting a new repo** — you do **not** need it before `rb-refresh` on an existing project. @@ -351,6 +368,43 @@ All are generated by `rb init`: `AGENTS.md` is the single behavioral rulebook, I --- +## Let another AI call RepoBrain (CLI, no MCP) + +The lightest way to let another LLM or agent use RepoBrain is the CLI — no long-running server, no protocol handshake. Any agent that can run a shell command can call: + +```bash +rb-ask "" --workspace /path/to/project --json +``` + +and read back a stable JSON object: + +```json +{ + "answer": "Auth is handled in engine/hub/auth.py …", + "sources": ["engine/hub/auth.py:12", "engine/hub/auth.py:44"], + "limitations": ["host-runner single-turn mode"], + "workspace": "/path/to/project", + "question": "" +} +``` + +- **Success** → the envelope above on stdout, exit code `0`. +- **Failure** → stdout stays empty; a `{"error": "..."}` object is written to stderr with a non-zero exit code, so your wrapper can branch on it without scraping text. + +**How agents discover this automatically:** `rb init` drops an `AGENTS.md` (and `CLAUDE.md` for Claude Code) into the project telling any agent to prefer `rb-ask` over manual grep/file search. Editors like Cursor, Windsurf, Codex, and Gemini CLI read those files, so they'll call `rb-ask` on their own once the project is initialized. + +**Zero API key:** `rb-ask` runs the same engine as everything else, so it honors a no-API-key local host runner. Put this in the project's `.env` (or run `rb-setup`) and the calling AI drives a CLI you're already logged into — no key changes hands: + +```bash +RB_HOST_RUNNER=generic +RB_HOST_COMMAND=trae-cli exec --cd {workspace} --sandbox read-only --skip-git-repo-check --ephemeral -o {output_file} +RB_HOST_OUTPUT_MODE=file +``` + +Prefer this over the MCP server (below) whenever the caller can shell out; reach for `rb-mcp` only for clients that speak MCP exclusively. + +--- + ## Advanced Features
diff --git a/README_CN.md b/README_CN.md index 859a7b68d..988c5e0a1 100644 --- a/README_CN.md +++ b/README_CN.md @@ -54,6 +54,12 @@ **与 Codex CLI 和 Claude Code 在三个真实 Python 仓库(`fastapi`、`requests`、`sqlmodel`)上做了 36 道题的三方对决——RepoBrain 事实题 99%、审计题 97%,事实题速度比 Codex 快 2.1×。** [查看对比](#三方对决repobrain-vs-codex-cli-vs-claude-code2026-05-09) +### 🧠 已经在用 AI IDE?让它帮你装 RepoBrain —— 无需 API key。 + +如果你在用 Trae / Cursor / Claude Code / Codex(且已登录),你**不用碰 pip,也不用 API key**。 +**把 [`AI_INSTALL.md`](AI_INSTALL.md) 整段贴给你的 AI 助手,说"照着它在这个项目里装好 RepoBrain"。** +它会探测你已登录的 CLI、配好零-key 后端、初始化项目并自测——之后你直接问它关于代码库的任何问题即可。 + ``` 传统做法: RepoBrain 做法: CLAUDE.md = 5000 行文档 Claude Code 调用 ask_project("auth 怎么工作的?") @@ -87,7 +93,7 @@ ### `rb-setup` —— 首次配置 -每个项目跑**一次**,在安装插件后立即执行。交互式选择 LLM 提供商(OpenAI / DeepSeek / Groq / 阿里灵积 / NVIDIA NIM / Ollama 本地 / 任意 OpenAI 兼容端点),然后在项目根目录写入 `.env`,包含 `OPENAI_BASE_URL`、`OPENAI_API_KEY`、`OPENAI_MODEL`、`RB_ASK_TIMEOUT_SECONDS`。也可以选择本地 Codex host-runner 实验模式,让无 API key 的 `rb-ask` 通过本机 `codex login` 运行。命令会把 `.env` 加入 `.gitignore`。如果已经有可用的 `.env` 可跳过。 +每个项目跑**一次**,在安装插件后立即执行。交互式向导会**先探测你本机已登录的无头 CLI**(Codex / Trae / Claude / Gemini),并把**免 API key 的本地 host-runner** 作为最省事的首选项——不用贴 key,RepoBrain 直接驱动你现有的 CLI 登录,`rb-ask` 和 `rb-refresh` 都能用。如果你更想用托管模型,向导同样提供 API-key 提供商(OpenAI / DeepSeek / Groq / 阿里灵积 / NVIDIA NIM / Ollama 本地 / 任意 OpenAI 兼容端点)。两种方式都会在项目根目录写入 `.env`——本地 CLI 写 `RB_HOST_RUNNER` + `RB_HOST_COMMAND`,提供商写 `OPENAI_BASE_URL` / `OPENAI_API_KEY` / `OPENAI_MODEL`——并把 `.env` 加入 `.gitignore`。如果已经有可用的 `.env` 可跳过。 ``` # Claude Code @@ -111,7 +117,7 @@ /rb-refresh quick ``` -耗时:小仓库几分钟,大仓库更久。需要先完成 `rb-setup`。完整 LLM refresh 仍需要 API key / OpenAI-compatible provider;本地 host-runner 模式下可用 `RB_REFRESH_SCAN_ONLY=1 rb-refresh --workspace .` 生成本地扫描知识产物。 +耗时:小仓库几分钟,大仓库更久。需要先完成 `rb-setup`。两种后端都能用:API key / OpenAI 兼容 provider 跑完整 LLM refresh;**本地 host-runner**(Codex / Trae / Claude / …)则通过你已登录的 CLI 跑无工具阶段(module 文档、`map.md`),并把工具/handoff 阶段(conventions、git insights)自动降级为确定性产物——全程无需 API key。只有当你想要"仅结构索引、无 LLM 叙述"的极速模式时,才加 `RB_REFRESH_SCAN_ONLY=1`。 ### `rb-ask` —— 路由问答 @@ -127,6 +133,15 @@ 需要已有知识库 —— 如果出现"无索引"或空答复,先跑 `rb-refresh`。 +**让别的 AI / 脚本来调用?** 加 `--json`,拿到稳定可解析的信封,而不是给人看的富文本——这是让任意能跑 shell 命令的 agent 调用 RepoBrain 的最轻量方式,**无需 MCP 服务**: + +```bash +rb-ask "认证逻辑是怎么实现的?" --workspace . --json +# → {"answer": "...", "sources": [...], "limitations": [...], "workspace": "...", "question": "..."} +``` + +出错时 `--json` 会保持 stdout 为空,把 `{"error": "..."}` 写到 stderr 并返回非零退出码,调用方无需正则去扒文本。它跑的是同一套引擎,所以既支持 API-key 提供商,也支持免 API key 的本地 host-runner。详见 [让别的 AI 调用 RepoBrain(CLI,免 MCP)](#让别的-ai-调用-repobraincli免-mcp)。 + ### `rb-init` —— 新仓库脚手架 基于 RepoBrain 模板创建**新**项目。两种模式:`quick`(快速脚手架、干净副本)和 `full`(在 quick 基础上加运行时 profile、`.env`、mission 文件、沙箱配置、可选 `git init`)。用于**开新仓库** —— 在已有项目上跑 `rb-refresh` 之前**不需要**先执行它。 @@ -152,7 +167,7 @@ # Claude Code(首次会话由 SessionStart hook 自动安装 rb CLI + Python 引擎) /plugin marketplace add study8677/repobrain /plugin install repobrain@repobrain -/repobrain:rb-setup # 交互式:选 LLM 提供商、贴 API key,自动写 .env +/repobrain:rb-setup # 交互式:用已登录的本地 CLI(Codex/Trae/Claude,免 key)或贴 API key,自动写 .env /repobrain:rb-refresh # 直接运行 rb-refresh;首次 refresh 会自动创建 .repobrain/ /repobrain:rb-ask "这个项目是怎么工作的?" # 直接运行 rb-ask @@ -342,6 +357,43 @@ Ask 管道采用**语义路径**:Router 读取 `map.md` → 选择模块 → --- +## 让别的 AI 调用 RepoBrain(CLI,免 MCP) + +让别的 LLM / agent 用上 RepoBrain,最轻的方式就是 CLI —— 没有常驻进程,没有协议握手。任何能跑一条 shell 命令的 agent 都可以直接调: + +```bash +rb-ask "<问题>" --workspace /path/to/project --json +``` + +然后读回一个稳定的 JSON 对象: + +```json +{ + "answer": "认证逻辑在 engine/hub/auth.py …", + "sources": ["engine/hub/auth.py:12", "engine/hub/auth.py:44"], + "limitations": ["host-runner 单轮模式"], + "workspace": "/path/to/project", + "question": "<问题>" +} +``` + +- **成功** → 上面这个信封打到 stdout,退出码 `0`。 +- **失败** → stdout 保持为空;`{"error": "..."}` 写到 stderr,退出码非零 —— 调用方可直接分支处理,无需扒文本。 + +**agent 怎么自动发现它:** `rb init` 会往项目里放 `AGENTS.md`(Claude Code 再加 `CLAUDE.md`),告诉任意 agent 遇到代码库问题优先用 `rb-ask` 而不是手动 grep / 读文件。Cursor、Windsurf、Codex、Gemini CLI 这些都会读这些文件,所以项目初始化后它们会自己去调 `rb-ask`。 + +**零 API key:** `rb-ask` 跑的是同一套引擎,因此同样支持免 API key 的本地 host-runner。在项目 `.env` 里写下面几行(或跑 `rb-setup`),调用方的 AI 就会驱动你本机已登录的 CLI,全程不经手 key: + +```bash +RB_HOST_RUNNER=generic +RB_HOST_COMMAND=trae-cli exec --cd {workspace} --sandbox read-only --skip-git-repo-check --ephemeral -o {output_file} +RB_HOST_OUTPUT_MODE=file +``` + +只要调用方能执行 shell 命令,就优先用这条 CLI 路径;只有面对**只认 MCP 协议**的客户端时,才用下面的 `rb-mcp`。 + +--- + ## 进阶功能
diff --git a/cli/pyproject.toml b/cli/pyproject.toml index 467f12649..5041e38b5 100644 --- a/cli/pyproject.toml +++ b/cli/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "hatchling.build" [project] name = "repobrain-cli" -version = "2.0.1" +version = "2.0.2" description = "Inject RepoBrain context files and offline helpers into any project." readme = "README.md" license = "MIT" diff --git a/cli/src/rb_cli/__init__.py b/cli/src/rb_cli/__init__.py index a77be7242..c2012217f 100644 --- a/cli/src/rb_cli/__init__.py +++ b/cli/src/rb_cli/__init__.py @@ -1,3 +1,3 @@ """RepoBrain CLI – inject cognitive architecture into any project.""" -__version__ = "2.0.1" +__version__ = "2.0.2" diff --git a/cli/src/rb_cli/cli.py b/cli/src/rb_cli/cli.py index 32b4d6e30..c8432c79a 100644 --- a/cli/src/rb_cli/cli.py +++ b/cli/src/rb_cli/cli.py @@ -451,10 +451,18 @@ def version_cmd() -> None: def ask_cmd( question: str = typer.Argument(..., help="Question about the project."), workspace: str = typer.Option(".", "--workspace", "-w", help="Project directory."), + json_output: bool = typer.Option( + False, + "--json", + help="Emit a machine-readable JSON envelope for programmatic callers.", + ), ) -> None: """Ask a question about the project (requires LLM).""" workspace_path = Path(workspace).resolve() - code = _run_hub(workspace_path, "ask", question) + args = ["ask", question] + if json_output: + args.append("--json") + code = _run_hub(workspace_path, *args) raise typer.Exit(code=code) diff --git a/cli/src/rb_cli/templates/.trae/rules/project_rules.md b/cli/src/rb_cli/templates/.trae/rules/project_rules.md new file mode 100644 index 000000000..7ece6177a --- /dev/null +++ b/cli/src/rb_cli/templates/.trae/rules/project_rules.md @@ -0,0 +1,32 @@ +# Trae bootstrap rules + +Authoritative behavior rules live in `AGENTS.md` at the project root. Read it +first, then load dynamic context from `.repobrain/` (`conventions.md`, +`structure.md`, `decisions/log.md`, `memory/`). + +## Use the RepoBrain knowledge hub + +For broad codebase questions — "where is X implemented", "how does X work", +architecture, dependency/impact analysis, onboarding — prefer running: + +```bash +rb-ask "" --workspace . --json +``` + +over grep / rg / manual file reading. It returns an answer grounded in real +source with file paths and line numbers. The `--json` form gives you a stable +`{answer, sources, limitations, workspace, question}` object; on failure stdout +stays empty and a `{"error": "..."}` object is written to stderr with a +non-zero exit code, so you can branch on it cleanly. + +You do **not** need to run `rb-refresh` by hand: `rb-ask` builds the knowledge +base itself on first use and rebuilds it when it drifts too far behind HEAD +(controlled by `RB_ASK_AUTO_REFRESH` in `.env`). Run `rb-refresh --workspace .` +explicitly only when you want to force a full rebuild. + +This works with an API key **or**, with no API key, a local host runner +(`RB_HOST_RUNNER` in `.env`) — including Trae itself — so you can query +RepoBrain without any key changing hands. + +Use direct file reads or rg only to verify exact lines after `rb-ask` points at +a file, for narrow symbol searches, or when `rb-ask` is unavailable. diff --git a/cli/src/rb_cli/templates/AGENTS.md b/cli/src/rb_cli/templates/AGENTS.md index 0f4f2ba20..60e1265b4 100644 --- a/cli/src/rb_cli/templates/AGENTS.md +++ b/cli/src/rb_cli/templates/AGENTS.md @@ -29,13 +29,37 @@ rb-ask "" --workspace . Use this before broad grep, rg, or file search when `.repobrain/` exists. -Run: +If you are an LLM or script calling RepoBrain programmatically (not a human +reading the terminal), add `--json` to get a stable, parseable envelope instead +of formatted prose — no need to scrape the text: ```bash -rb-refresh --workspace . +rb-ask "" --workspace . --json +# → {"answer": "...", "sources": [...], "limitations": [...], +# "workspace": "...", "question": "..."} ``` -when `.repobrain/` is missing, stale, or after significant code changes. +On failure with `--json`, stdout stays empty and a `{"error": "..."}` object is +written to stderr with a non-zero exit code, so callers can branch on it cleanly. + +This CLI is the lightweight way to let any agent that can run shell commands +query RepoBrain — no long-running MCP server required. (An MCP server, `rb-mcp`, +also exists for MCP-only clients, but the CLI is preferred when you can shell +out.) Both paths run the same engine, so they work with an API-key provider or, +with no API key, a local host runner (`RB_HOST_RUNNER` in `.env`) that drives a +CLI you are already logged into (Codex / Trae / Claude / …). + +You normally do **not** need to run `rb-refresh` yourself: `rb-ask` keeps its +own knowledge base current. It builds the base automatically on first use (when +`.repobrain/` is missing) and rebuilds it when it drifts too far behind HEAD. +This is governed by `RB_ASK_AUTO_REFRESH` in `.env` (`stale` = first-run + +drift, the default; `first-only`; or `off`). + +Run this explicitly only to force a full rebuild: + +```bash +rb-refresh --workspace . +``` Use direct file reads or rg only for: diff --git a/commands/rb-ask.md b/commands/rb-ask.md index 6431273ed..8c7da66c4 100644 --- a/commands/rb-ask.md +++ b/commands/rb-ask.md @@ -15,14 +15,16 @@ Use Bash: RB_ASK_TIMEOUT_SECONDS="${RB_ASK_TIMEOUT_SECONDS:-120}" rb-ask "$ARGUMENTS" --workspace "$PWD" ``` -If `.env` sets `RB_HOST_RUNNER=codex`, the same command uses the user's local -Codex CLI login for `rb-ask` instead of an API key. This host-runner mode is -ask-only; refresh should use scan-only artifacts or a configured `OPENAI_*` -provider. - -如果 `.env` 设置了 `RB_HOST_RUNNER=codex`,同一个命令会通过用户本机 Codex CLI -登录运行 `rb-ask`,不走 API key。这个 host-runner 模式只支持 ask;refresh -应使用 scan-only 产物或已配置的 `OPENAI_*` provider。 +If `.env` sets `RB_HOST_RUNNER` (`codex` or `generic`), the same command drives +the user's local logged-in CLI (Codex / Trae / Claude / …) for `rb-ask` instead +of an API key. The same host-runner backend also powers `rb-refresh` (tool-free +stages run through the CLI; conventions and git insights degrade to +deterministic output). + +如果 `.env` 设置了 `RB_HOST_RUNNER`(`codex` 或 `generic`),同一个命令会通过 +用户本机已登录的 CLI(Codex / Trae / Claude / …)运行 `rb-ask`,不走 API key。 +同一套 host-runner 后端也支撑 `rb-refresh`(无工具阶段走 CLI,conventions 与 +git insights 降级为确定性产物)。 使用 Bash: diff --git a/commands/rb-setup.md b/commands/rb-setup.md index e569119ff..5ea63fba0 100644 --- a/commands/rb-setup.md +++ b/commands/rb-setup.md @@ -1,10 +1,20 @@ --- -description: First-time setup. Configure the LLM API key or local Codex host runner RepoBrain uses for codebase Q&A. / 首次 setup,配置 RepoBrain 代码问答所需的 LLM API key 或本地 Codex host runner。 +description: First-time setup. Configure the LLM API key, or a no-API-key local host runner (Codex / Trae / Claude / any headless CLI) that RepoBrain uses for codebase Q&A and refresh. / 首次 setup,配置 RepoBrain 代码问答与 refresh 所需的 LLM API key,或无需 API key 的本地 host runner(Codex / Trae / Claude / 任意无头 CLI)。 --- -You are running first-time setup for the RepoBrain plugin. The user just installed the plugin and needs either an LLM API key or the explicit local Codex host-runner mode configured before the ask commands will work (`/repobrain:rb-ask` in Claude Code; `/rb-ask` in Codex CLI). Full LLM refresh still requires an API key; no-key users can run scan-only refresh. Goal: write a `.env` file at the current workspace root. +You are running first-time setup for the RepoBrain plugin. The user just installed the plugin and needs an LLM backend configured before the ask/refresh commands will work (`/repobrain:rb-ask` in Claude Code; `/rb-ask` in Codex CLI). There are two families of backends: -你正在执行 RepoBrain 插件的首次 setup。用户刚安装插件,需要先配置 LLM API key,或显式启用本地 Codex host-runner 模式,ask 命令才能正常工作(Claude Code 内为 `/repobrain:rb-ask`;Codex CLI 内为 `/rb-ask`)。完整 LLM refresh 仍需要 API key;无 key 用户可以运行 scan-only refresh。目标是在当前工作区根目录写入 `.env` 文件。 +1. **API-key providers** (OpenAI-compatible) — write `OPENAI_*` keys. +2. **Local host runners (no API key)** — drive a headless CLI the user already has logged in (Codex, Trae, Claude, or any command that answers a prompt on stdin). This covers both `rb-ask` **and** `rb-refresh`: refresh runs its tool-free stages (module docs, map) through the host runner and automatically falls back to deterministic output for tool/handoff stages (conventions, git insights). + +Goal: write a `.env` file at the current workspace root. + +你正在执行 RepoBrain 插件的首次 setup。用户刚安装插件,需要先配置一个 LLM 后端,ask/refresh 命令才能正常工作(Claude Code 内为 `/repobrain:rb-ask`;Codex CLI 内为 `/rb-ask`)。后端分两类: + +1. **API-key 提供商**(OpenAI 兼容)—— 写入 `OPENAI_*` 配置。 +2. **本地 host runner(无需 API key)**—— 驱动用户本机已登录的无头 CLI(Codex、Trae、Claude,或任意能在命令行吃 prompt、吐文本的命令)。这条路同时支持 `rb-ask` **和** `rb-refresh`:refresh 的无工具阶段(module 文档、map)走 host runner,工具/handoff 阶段(conventions、git insights)会自动降级为确定性产物。 + +目标是在当前工作区根目录写入 `.env` 文件。 ## Step 1 — Detect existing config / 步骤 1 —— 检测已有配置 @@ -12,23 +22,38 @@ Read `.env` at the workspace root if it exists. If `OPENAI_API_KEY` or `RB_HOST_ 如果工作区根目录已有 `.env`,先读取它。如果已经设置了 `OPENAI_API_KEY` 或 `RB_HOST_RUNNER`,询问用户是否覆盖 RepoBrain 的 LLM/host-runner 配置。若用户选择不覆盖,确认“already configured / 已配置”并停止。 -## Step 2 — Ask which LLM provider (use AskUserQuestion) / 步骤 2 —— 询问 LLM 提供商(使用 AskUserQuestion) +## Step 2 — Ask which backend (use AskUserQuestion) / 步骤 2 —— 询问后端(使用 AskUserQuestion) + +First, detect which local headless CLIs are available so you only offer runners that can actually work. Run these checks (ignore ones that error): + +先探测本机可用的无头 CLI,只向用户提供真正能用的 runner。运行以下检查(报错的忽略即可): + +- `command -v codex` and, if present, `codex login status` +- `command -v trae-cli` and, if present, `trae-cli login status` +- `command -v claude` (Claude Code; logged-in state is implicit) +- `command -v gemini`, `command -v ollama` -Present these options: +Present these options. **List the detected local CLIs first** (they need no API key), then the API-key providers: -向用户展示以下选项: +向用户展示以下选项。**优先列出探测到的本地 CLI**(无需 API key),再列 API-key 提供商: +- **本地 CLI(无 API key)** — pick this if `codex` / `trae-cli` / `claude` / `gemini` was detected. Drives your already-logged-in CLI for both `rb-ask` and `rb-refresh`. - **OpenAI** — gpt-4o-mini / gpt-4o - **DeepSeek** — cheap, strong on code - **Groq** — fast, free tier - **阿里灵积 (DashScope)** — qwen 系列 - **NVIDIA NIM** — generous free tier -- **Ollama 本地** — no key needed, runs locally -- **Codex CLI 本地实验模式** — no API key, uses the user's local `codex login`; supports `rb-ask` only +- **Ollama 本地** — no key needed, runs a local model server - **其他 OpenAI 兼容端点** — custom URL +If the user picks **本地 CLI(无 API key)**, ask a follow-up with the concrete detected runners (e.g. Codex / Trae / Claude) so they choose exactly one. + +如果用户选择 **本地 CLI(无 API key)**,再追问一次,列出具体探测到的 runner(如 Codex / Trae / Claude),让用户选定其中一个。 + ## Step 3 — Collect URL / key / model / 步骤 3 —— 收集 URL / key / model +### 3a — API-key providers / API-key 提供商 + Use this table to set the URL and suggest a model based on the provider: 根据用户选择的提供商,使用下表设置 URL 并建议模型: @@ -41,16 +66,38 @@ Use this table to set the URL and suggest a model based on the provider: | 阿里灵积 | `https://dashscope.aliyuncs.com/compatible-mode/v1` | `qwen-max` | | NVIDIA NIM | `https://integrate.api.nvidia.com/v1` | `meta/llama-3.3-70b-instruct` | | Ollama 本地 | `http://localhost:11434/v1` | `llama3.2` (key can be `ollama`) | -| Codex CLI 本地实验模式 | not used | `gpt-5.3-codex-spark` via `RB_HOST_MODEL` | | 其他 | ask the user | ask the user | For non-Ollama providers ask the user to paste their key. For Ollama use `OPENAI_API_KEY=ollama` (the engine requires the field to be non-empty). 非 Ollama 提供商需要让用户粘贴 API key。Ollama 使用 `OPENAI_API_KEY=ollama`(engine 要求该字段非空)。 -For Codex CLI local experimental mode, first check that `codex login status` reports a ChatGPT login. Do not ask for an API key and do not write a fake `OPENAI_API_KEY`. +### 3b — Local host runners (no API key) / 本地 host runner(无 API key) + +For a local runner, first **verify the CLI is logged in** — do NOT ask for an API key and do NOT write a fake `OPENAI_API_KEY`. If the login check fails, tell the user to log in first and stop: + +选了本地 runner 时,先**确认该 CLI 已登录**——不要询问 API key,也不要写假的 `OPENAI_API_KEY`。若登录检查失败,提示用户先登录并停止: + +| Runner | Login check / 登录检查 | `RB_HOST_RUNNER` | Notes / 说明 | +|---|---|---|---| +| Codex | `codex login status` must report a ChatGPT login | `codex` | Built-in preset; model via `RB_HOST_MODEL` | +| Trae | `trae-cli login status` must report logged in | `generic` | Set `RB_HOST_COMMAND` (below) | +| Claude | `claude` present (login is implicit) | `generic` | Set `RB_HOST_COMMAND` (below) | +| 其他 CLI | ask the user how to run it headlessly | `generic` | Set `RB_HOST_COMMAND` (below) | + +For **Codex**, no command template is needed — it is a built-in preset. For every **generic** runner, write an `RB_HOST_COMMAND` template. Use these verified templates (all deliver the prompt on **stdin** and read the answer from `{output_file}`, so `RB_HOST_OUTPUT_MODE=file`): -Codex CLI 本地实验模式需要先确认 `codex login status` 显示已用 ChatGPT 登录。不要询问 API key,也不要写假的 `OPENAI_API_KEY`。 +**Codex** 无需命令模板(内置预设)。所有 **generic** runner 都需要写 `RB_HOST_COMMAND` 模板。使用下列已验证的模板(都通过 **stdin** 传入 prompt,从 `{output_file}` 读回答案,因此 `RB_HOST_OUTPUT_MODE=file`): + +| Runner | `RB_HOST_COMMAND` | +|---|---| +| Trae | `trae-cli exec --cd {workspace} --sandbox read-only --skip-git-repo-check --ephemeral -o {output_file}` | +| Claude | `claude -p --add-dir {workspace}` (uses `RB_HOST_OUTPUT_MODE=stdout`) | +| 其他 | Ask the user for a command that reads a prompt on stdin and prints the answer. Add `-o {output_file}` if the CLI supports writing its final message to a file; otherwise use stdout mode. | + +Do NOT put `{prompt_file}` in the template — omitting it makes RepoBrain feed the prompt on stdin, which is the most portable path. Only add `{prompt_file}` if a CLI cannot read stdin. + +模板里**不要**写 `{prompt_file}`——省略它 RepoBrain 会自动把 prompt 喂给 stdin,这是最通用的方式。只有当某个 CLI 无法读 stdin 时,才加 `{prompt_file}`。 ## Step 4 — Write `.env` / 步骤 4 —— 写入 `.env` @@ -69,19 +116,35 @@ OPENAI_MODEL= RB_ASK_TIMEOUT_SECONDS=120 ``` -For Codex CLI local experimental mode: +For the **Codex** local runner (built-in preset, no command template): -Codex CLI 本地实验模式写入: +**Codex** 本地 runner(内置预设,无需命令模板)写入: ``` RB_HOST_RUNNER=codex RB_HOST_MODEL=gpt-5.3-codex-spark RB_HOST_TIMEOUT_SECONDS=240 RB_HOST_MAX_CONTEXT_CHARS=60000 -RB_REFRESH_SCAN_ONLY=1 RB_ASK_TIMEOUT_SECONDS=120 ``` +For a **generic** local runner (Trae / Claude / other) — substitute the `RB_HOST_COMMAND` and `RB_HOST_OUTPUT_MODE` chosen in Step 3b: + +**generic** 本地 runner(Trae / Claude / 其他)写入 —— 代入步骤 3b 选定的 `RB_HOST_COMMAND` 与 `RB_HOST_OUTPUT_MODE`: + +``` +RB_HOST_RUNNER=generic +RB_HOST_COMMAND= +RB_HOST_OUTPUT_MODE= +RB_HOST_TIMEOUT_SECONDS=240 +RB_HOST_MAX_CONTEXT_CHARS=60000 +RB_ASK_TIMEOUT_SECONDS=120 +``` + +Do **not** write `RB_REFRESH_SCAN_ONLY=1` by default: host-runner refresh now runs the module/map LLM stages and gracefully degrades the tool/handoff stages. Only add `RB_REFRESH_SCAN_ONLY=1` if the user explicitly wants a fast structure-only index with no LLM narration. Do not write any `OPENAI_*` keys for a local runner. + +默认**不要**写 `RB_REFRESH_SCAN_ONLY=1`:host-runner 的 refresh 现在会跑 module/map 的 LLM 阶段,并对工具/handoff 阶段自动降级。只有当用户明确想要"仅结构索引、无 LLM 叙述"的快速模式时,才加 `RB_REFRESH_SCAN_ONLY=1`。本地 runner 不要写任何 `OPENAI_*` 配置。 + If `.env` already existed and the user opted to overwrite, replace only the relevant keys above; preserve any other lines. 如果 `.env` 已存在且用户选择覆盖,只替换上面相关 key;保留其他行。 @@ -109,19 +172,20 @@ Next / 下一步: 询问任何关于代码库的问题。 ``` -If configured for Codex CLI local experimental mode, print this instead: +If configured for a local host runner (Codex / Trae / Claude / other), print this instead — fill in the runner name: -如果配置的是 Codex CLI 本地实验模式,改为输出: +如果配置的是本地 host runner(Codex / Trae / Claude / 其他),改为输出(填入 runner 名称): ``` -✅ RepoBrain is configured for local Codex host-runner mode. -✅ RepoBrain 已配置为本地 Codex host-runner 模式。 +✅ RepoBrain is configured for local host-runner mode (). +✅ RepoBrain 已配置为本地 host-runner 模式()。 Next / 下一步: 1. /rb-refresh - 无 API key 时生成本地扫描知识产物。 + 通过本机 构建知识库;module 文档与 map 由 生成, + conventions 与 git insights 自动降级为确定性产物。 2. /rb-ask - 通过本机 Codex 登录询问当前代码库。 + 通过本机 询问当前代码库(零 API key)。 ``` Do NOT call MCP tools from this command. The refresh and ask slash commands use the CLI (`rb-refresh` / `rb-ask`) directly and will read the `.env` file on each run. diff --git a/engine/repobrain_engine/_cli_entry.py b/engine/repobrain_engine/_cli_entry.py index a4a20103d..23aa4fbe8 100644 --- a/engine/repobrain_engine/_cli_entry.py +++ b/engine/repobrain_engine/_cli_entry.py @@ -8,7 +8,9 @@ from __future__ import annotations import argparse +import json import os +import re import sys import traceback from pathlib import Path @@ -41,6 +43,68 @@ def _run_ask_pipeline(workspace: Path, question: str) -> str: return asyncio.run(ask_pipeline(workspace, question)) +def _split_answer_sections(answer_text: str) -> dict[str, object]: + """Split a rendered answer into ``{answer, sources, limitations}``. + + The pipeline returns a single Markdown string. The host-runner path + appends ``Sources:`` / ``Limitations:`` blocks (see + :meth:`HostRunnerAnswer.to_markdown`); the LLM/structured paths return + free-form prose with no such headers. This best-effort splitter peels off + those trailing blocks when present so programmatic callers get structured + fields, and otherwise leaves the whole text as ``answer`` with empty lists. + + Args: + answer_text: The rendered answer string from ``ask_pipeline``. + + Returns: + A dict with ``answer`` (str), ``sources`` (list[str]), and + ``limitations`` (list[str]). + """ + sources: list[str] = [] + limitations: list[str] = [] + + # Match a trailing "Limitations:" block, then a trailing "Sources:" block. + # Order matters: strip Limitations (which follows Sources) first so the + # Sources regex then sees a clean tail. + def _pop_block(text: str, header: str) -> tuple[str, list[str]]: + pattern = re.compile( + rf"\n\n{re.escape(header)}:\n((?:- .*(?:\n|$))+)\Z" + ) + match = pattern.search(text) + if not match: + return text, [] + items = [ + line[2:].strip() + for line in match.group(1).splitlines() + if line.startswith("- ") + ] + return text[: match.start()], [item for item in items if item] + + remaining, limitations = _pop_block(answer_text, "Limitations") + remaining, sources = _pop_block(remaining, "Sources") + + return { + "answer": remaining.strip(), + "sources": sources, + "limitations": limitations, + } + + +def _emit_ask_json(workspace: Path, question: str, answer_text: str) -> None: + """Print the ask result as a stable JSON envelope on stdout. + + Guarantees a machine-parseable object so LLM tool wrappers never have to + scrape human-formatted text: + + {"answer": ..., "sources": [...], "limitations": [...], + "workspace": ..., "question": ...} + """ + payload = _split_answer_sections(answer_text) + payload["workspace"] = str(workspace) + payload["question"] = question + print(json.dumps(payload, ensure_ascii=False, indent=2)) + + def _run_refresh_pipeline(workspace: Path, *, quick: bool, failed_only: bool): """Run the refresh pipeline for CLI entry points.""" import asyncio @@ -129,19 +193,46 @@ def ask_main(argv: Sequence[str] | None = None) -> None: ) parser.add_argument("question", help="Natural language question about the project") parser.add_argument("--workspace", default=".", help="Project root (default: cwd)") + parser.add_argument( + "--json", + action="store_true", + help="Emit a machine-readable JSON envelope " + "{answer, sources, limitations, workspace, question} instead of " + "human-formatted text. Errors are also emitted as JSON on stderr. " + "Use this when an LLM or script calls rb-ask programmatically.", + ) args = _parse_args(parser, argv) workspace = Path(args.workspace).resolve() os.environ["WORKSPACE_PATH"] = str(workspace) try: - print(_run_ask_pipeline(workspace, args.question)) + answer_text = _run_ask_pipeline(workspace, args.question) + if args.json: + _emit_ask_json(workspace, args.question, answer_text) + else: + print(answer_text) except KeyboardInterrupt: sys.exit(130) except ValueError as exc: - print(f"Error: {exc}", file=sys.stderr) + if args.json: + print( + json.dumps({"error": str(exc)}, ensure_ascii=False), + file=sys.stderr, + ) + else: + print(f"Error: {exc}", file=sys.stderr) sys.exit(1) except Exception as exc: # noqa: BLE001 - CLI boundary formats unknown failures + if args.json: + print( + json.dumps( + {"error": f"{exc.__class__.__name__}: {_one_line_message(exc)}"}, + ensure_ascii=False, + ), + file=sys.stderr, + ) + sys.exit(1) _handle_unexpected_cli_exception(exc) diff --git a/engine/repobrain_engine/config.py b/engine/repobrain_engine/config.py index 0fda5ce0d..6252d5907 100644 --- a/engine/repobrain_engine/config.py +++ b/engine/repobrain_engine/config.py @@ -65,11 +65,23 @@ class Settings(BaseSettings): # Local host runner (experimental, no API key) Configuration RB_HOST_RUNNER: str = Field( default="", - description="Experimental local host runner for rb-ask, e.g. 'codex'.", + description="Experimental local host runner for rb-ask, e.g. 'codex' or 'generic'.", ) RB_HOST_MODEL: str = Field( default="gpt-5.3-codex-spark", - description="Model passed to the local host runner.", + description="Model passed to the local host runner (used by the 'codex' runner).", + ) + RB_HOST_COMMAND: str = Field( + default="", + description="Command template for RB_HOST_RUNNER=generic. Supports placeholders " + "{prompt_file}, {schema_file}, {output_file}, {workspace}. Example: " + "'trae exec --prompt {prompt_file}'. When {prompt_file} is omitted, the prompt " + "is sent on stdin.", + ) + RB_HOST_OUTPUT_MODE: str = Field( + default="file", + description="Where the generic host runner reads its JSON answer from: " + "'file' (the {output_file}) or 'stdout'.", ) RB_HOST_TIMEOUT_SECONDS: float = Field( default=240.0, @@ -84,6 +96,21 @@ class Settings(BaseSettings): description="Run refresh without LLM analysis and write scan artifacts only.", ) + # Auto-refresh gate for rb-ask (let the CLI refresh itself instead of + # relying on an agent to notice and run rb-refresh manually). + RB_ASK_AUTO_REFRESH: str = Field( + default="stale", + description="When rb-ask should build/rebuild the knowledge base on its " + "own: 'off' (never), 'first-only' (only when .repobrain is missing), or " + "'stale' (missing OR more than RB_ASK_AUTO_REFRESH_LAG commits behind " + "HEAD). Default 'stale' covers both first run and drift.", + ) + RB_ASK_AUTO_REFRESH_LAG: int = Field( + default=20, + description="Commit lag past which 'stale' mode triggers an auto-refresh " + "before answering. Ignored when RB_ASK_AUTO_REFRESH is 'off'/'first-only'.", + ) + # Memory Configuration MEMORY_FILE: str = "memory/agent_memory.md" MEMORY_SUMMARY_FILE: str = "memory/agent_summary.md" diff --git a/engine/repobrain_engine/hub/agents.py b/engine/repobrain_engine/hub/agents.py index d4c3c1466..e5ad1fd3b 100644 --- a/engine/repobrain_engine/hub/agents.py +++ b/engine/repobrain_engine/hub/agents.py @@ -11,19 +11,21 @@ from __future__ import annotations from pathlib import Path -from typing import TYPE_CHECKING, Optional +from typing import TYPE_CHECKING, Optional, Union if TYPE_CHECKING: from repobrain_engine.config import Settings + from repobrain_engine.hub.host_runner import HostRunnerModel -def create_model(settings: "Settings") -> str: - """Resolve an LLM model identifier from settings. +def create_model(settings: "Settings") -> "Union[str, HostRunnerModel]": + """Resolve an LLM model for the Agent SDK from settings. Priority: 1. OPENAI_BASE_URL (any key) → litellm/openai/ (custom endpoint) 2. OPENAI_API_KEY (no base) → (standard OpenAI) - 3. None → raise ValueError + 3. RB_HOST_RUNNER (no key) → HostRunnerModel (local CLI, no API key) + 4. None → raise ValueError When a custom OPENAI_BASE_URL is provided (e.g. NVIDIA, Ollama), the model is routed through litellm so that the Agent SDK can reach the @@ -32,11 +34,17 @@ def create_model(settings: "Settings") -> str: *current* settings always take effect — avoiding first-caller-wins bugs in long-lived processes. + A configured API backend always wins so users who *have* keys keep the + full-capability refresh (tool-using / handoff swarms). Only when no + OpenAI backend is present do we fall back to a local host runner, which + drives the tool-free stages of refresh without an API key. + Args: settings: Application settings. Returns: - A model string suitable for openai-agents[litellm]. + A model string suitable for openai-agents[litellm], or a + :class:`HostRunnerModel` instance backed by a local CLI. Raises: ValueError: When no LLM backend is configured. @@ -55,9 +63,29 @@ def create_model(settings: "Settings") -> str: if settings.OPENAI_API_KEY: return settings.OPENAI_MODEL + # No API key configured — drive a local headless CLI if requested. + from repobrain_engine.hub.host_runner import ( + HostRunnerModel, + is_host_runner_enabled, + ) + + # Use getattr so callers passing a minimal settings stub (older tests, + # embedders) without the RB_HOST_* fields still get the ValueError path. + host_runner = getattr(settings, "RB_HOST_RUNNER", "") + if is_host_runner_enabled(host_runner): + return HostRunnerModel( + runner=host_runner, + workspace=settings.project_root_path, + model=getattr(settings, "RB_HOST_MODEL", None), + command=getattr(settings, "RB_HOST_COMMAND", None), + output_mode=getattr(settings, "RB_HOST_OUTPUT_MODE", None), + timeout_seconds=getattr(settings, "RB_HOST_TIMEOUT_SECONDS", None), + ) + raise ValueError( "No LLM configured. Run rb-setup or set OPENAI_BASE_URL, " - "OPENAI_API_KEY, and OPENAI_MODEL in .env" + "OPENAI_API_KEY, and OPENAI_MODEL in .env — or set RB_HOST_RUNNER " + "(codex/generic) to drive a local CLI without an API key." ) @@ -184,6 +212,48 @@ def build_refresh_swarm(model: str): return scan_analyst +_SINGLE_TURN_CONVENTION_INSTRUCTIONS = """\ +You are a code analyst and technical writer. + +Given a project scan report, analyze the codebase and write a concise +conventions document in a **single pass** (no handoffs, no tools). Cover: +- Primary language(s) and framework(s), with concrete evidence +- Project directory structure and organization patterns +- Code style observations (naming, structure, idioms) +- Testing approach, framework, and coverage indicators +- CI/CD setup, build system, and container configuration + +Be specific — cite file counts, directory names, and concrete config files +you observe, not vague generalities. + +Keep it under 300 words. Output ONLY the Markdown content, no preamble, +no commentary. Start directly with a heading. +""" + + +def build_single_turn_convention_agent(model): + """Build a tool-free, single-turn conventions agent (no handoffs). + + This collapses the 3-hop ScanAnalyst → ArchitectureReviewer → + ConventionWriter handoff chain into one agent so it can be driven by a + :class:`~repobrain_engine.hub.host_runner.HostRunnerModel`, which only + supports single-turn, tool-free, handoff-free generation. + + Args: + model: Model identifier string or a ``Model`` instance. + + Returns: + A single Agent that emits the conventions Markdown in one turn. + """ + Agent = _import_agent() + return Agent( + name="ConventionWriterSingleTurn", + instructions=_SINGLE_TURN_CONVENTION_INSTRUCTIONS, + model=model, + **_get_model_settings_kwargs(), + ) + + # --------------------------------------------------------------------------- # Ask Swarm — Dynamic Module-based Router-Worker pattern # --------------------------------------------------------------------------- diff --git a/engine/repobrain_engine/hub/ask_pipeline.py b/engine/repobrain_engine/hub/ask_pipeline.py index 7d82a3786..6a54c6ba5 100644 --- a/engine/repobrain_engine/hub/ask_pipeline.py +++ b/engine/repobrain_engine/hub/ask_pipeline.py @@ -191,18 +191,22 @@ async def ask_pipeline(workspace: Path, question: str) -> str: """ from repobrain_engine.config import get_settings from repobrain_engine.hub.host_runner import ( + SUPPORTED_HOST_RUNNERS, HostRunnerError, is_host_runner_enabled, normalize_host_runner_name, ) settings = get_settings() + await _maybe_auto_refresh(workspace, settings) + host_runner = normalize_host_runner_name(settings.RB_HOST_RUNNER) if host_runner: if not is_host_runner_enabled(host_runner): + supported = ", ".join(sorted(SUPPORTED_HOST_RUNNERS)) raise HostRunnerError( f"Unsupported RB_HOST_RUNNER={settings.RB_HOST_RUNNER!r}. " - "Supported values: codex." + f"Supported values: {supported}." ) answer = await _ask_with_host_runner(workspace, question, settings) return _prepend_workspace_health_notices(workspace, answer) @@ -226,6 +230,85 @@ async def _once() -> str: return _prepend_workspace_health_notices(workspace, answer) +#: Guard against re-entrancy: refresh_pipeline may itself trigger code paths +#: that reach ask_pipeline. We never want an auto-refresh to recurse. +_AUTO_REFRESH_IN_PROGRESS = False + + +def _should_auto_refresh(workspace: Path, settings) -> str | None: + """Decide whether rb-ask should refresh itself before answering. + + Lets the CLI keep its own knowledge base current instead of relying on an + agent to notice staleness and run ``rb-refresh`` by hand. Any tool that + calls ``rb-ask`` then inherits auto-refresh for free. + + Args: + workspace: Project root directory. + settings: Loaded application settings. + + Returns: + A short human-readable reason string when a refresh should run, or + ``None`` to answer against existing (or absent) artifacts as-is. + """ + mode = str(getattr(settings, "RB_ASK_AUTO_REFRESH", "stale")).strip().lower() + if mode in {"off", "0", "false", "no", ""}: + return None + + if not _structured_artifacts_available(workspace): + return "no knowledge base found" + + if mode == "first-only": + return None + + # mode == "stale" (or any other truthy value): also refresh on drift. + lag = _get_refresh_commit_lag(workspace) + threshold = int(getattr(settings, "RB_ASK_AUTO_REFRESH_LAG", 20)) + if lag is not None and lag > threshold: + return f"knowledge base is {lag} commits behind HEAD" + return None + + +async def _maybe_auto_refresh(workspace: Path, settings) -> None: + """Run ``refresh_pipeline`` in-process when the gate says the KB is stale. + + Best-effort: a failed auto-refresh never blocks the answer — rb-ask then + proceeds against whatever artifacts exist (possibly none), exactly as + before this gate was added. + + Args: + workspace: Project root directory. + settings: Loaded application settings. + """ + global _AUTO_REFRESH_IN_PROGRESS + if _AUTO_REFRESH_IN_PROGRESS: + return + + reason = _should_auto_refresh(workspace, settings) + if reason is None: + return + + from repobrain_engine.hub.refresh_pipeline import refresh_pipeline + + print( + f"[auto-refresh] {reason}; building knowledge base " + "(set RB_ASK_AUTO_REFRESH=off to disable)...", + file=sys.stderr, + ) + _AUTO_REFRESH_IN_PROGRESS = True + try: + # quick=True keeps the pre-answer refresh light; a full rebuild is + # still available via an explicit `rb-refresh`. + await refresh_pipeline(workspace, quick=True) + except Exception as exc: # noqa: BLE001 - never let refresh block the answer + print( + f"[auto-refresh] skipped ({exc.__class__.__name__}: {exc}); " + "answering with existing knowledge.", + file=sys.stderr, + ) + finally: + _AUTO_REFRESH_IN_PROGRESS = False + + async def _ask_pipeline_once(workspace: Path, question: str) -> str: """Run one ask attempt: structured path first, then legacy swarm.""" from agents import set_tracing_disabled @@ -279,6 +362,8 @@ async def _ask_with_host_runner(workspace: Path, question: str, settings) -> str retrieval_evidence=retrieval_evidence, graph_context=graph_context, model=settings.RB_HOST_MODEL, + command=settings.RB_HOST_COMMAND, + output_mode=settings.RB_HOST_OUTPUT_MODE, timeout_seconds=settings.RB_HOST_TIMEOUT_SECONDS, max_context_chars=settings.RB_HOST_MAX_CONTEXT_CHARS, ) diff --git a/engine/repobrain_engine/hub/host_runner.py b/engine/repobrain_engine/hub/host_runner.py index d2b7b5330..4d825d0f9 100644 --- a/engine/repobrain_engine/hub/host_runner.py +++ b/engine/repobrain_engine/hub/host_runner.py @@ -1,9 +1,19 @@ -"""Local host-backed runners for experimental no-API-key workflows.""" +"""Local host-backed runners for experimental no-API-key workflows. + +Two runners are supported for ``rb-ask``: + +* ``codex`` — drive the user's local ``codex exec`` login (built-in preset). +* ``generic`` — drive *any* headless agent CLI (Trae, Gemini CLI, Claude Code, + or any command that answers a prompt on the command line) via the + ``RB_HOST_COMMAND`` template. This makes the host-runner mechanism portable + across hosts instead of being hard-wired to Codex. +""" from __future__ import annotations import json import os import re +import shlex import shutil import subprocess import tempfile @@ -16,6 +26,20 @@ DEFAULT_HOST_TIMEOUT_SECONDS = 240.0 DEFAULT_HOST_MAX_CONTEXT_CHARS = 60000 +#: Host runners recognized by ``rb-ask``. ``codex`` is a built-in preset; +#: ``generic`` is configured through ``RB_HOST_COMMAND``. +SUPPORTED_HOST_RUNNERS = frozenset({"codex", "generic"}) + +#: Placeholders substituted into ``RB_HOST_COMMAND`` for the generic runner. +GENERIC_PROMPT_FILE_PLACEHOLDER = "{prompt_file}" +GENERIC_SCHEMA_FILE_PLACEHOLDER = "{schema_file}" +GENERIC_OUTPUT_FILE_PLACEHOLDER = "{output_file}" +GENERIC_WORKSPACE_PLACEHOLDER = "{workspace}" + +#: How the generic runner recovers its answer: from ``{output_file}`` or stdout. +GENERIC_OUTPUT_MODES = frozenset({"file", "stdout"}) +DEFAULT_GENERIC_OUTPUT_MODE = "file" + class HostRunnerError(ValueError): """Raised when a local host runner cannot produce an answer.""" @@ -51,7 +75,17 @@ def normalize_host_runner_name(value: str | None) -> str: def is_host_runner_enabled(value: str | None) -> bool: """Return whether a supported local host runner was requested.""" - return normalize_host_runner_name(value) in {"codex"} + return normalize_host_runner_name(value) in SUPPORTED_HOST_RUNNERS + + +def is_host_runner_model(model: object) -> bool: + """Return whether ``model`` is a :class:`HostRunnerModel` instance. + + Used by ``rb-refresh`` to detect the no-API-key path and pre-emptively + route tool-using / handoff stages to deterministic fallbacks instead of + letting them fail at ``get_response``. + """ + return isinstance(model, HostRunnerModel) async def run_host_runner( @@ -63,24 +97,43 @@ async def run_host_runner( retrieval_evidence: str | None = None, graph_context: str | None = None, model: str | None = None, + command: str | None = None, + output_mode: str | None = None, timeout_seconds: float | None = None, max_context_chars: int | None = None, ) -> str: """Run the configured local host runner and return Markdown output.""" runner_name = normalize_host_runner_name(runner) - if runner_name != "codex": - raise HostRunnerError(f"Unsupported host runner: {runner or ''}") + if runner_name not in SUPPORTED_HOST_RUNNERS: + supported = ", ".join(sorted(SUPPORTED_HOST_RUNNERS)) + raise HostRunnerError( + f"Unsupported host runner: {runner or ''}. " + f"Supported values: {supported}." + ) - answer = await run_codex_host_runner( - workspace=workspace, - question=question, - context=context, - retrieval_evidence=retrieval_evidence, - graph_context=graph_context, - model=model, - timeout_seconds=timeout_seconds, - max_context_chars=max_context_chars, - ) + if runner_name == "codex": + answer = await run_codex_host_runner( + workspace=workspace, + question=question, + context=context, + retrieval_evidence=retrieval_evidence, + graph_context=graph_context, + model=model, + timeout_seconds=timeout_seconds, + max_context_chars=max_context_chars, + ) + else: # runner_name == "generic" + answer = await run_generic_host_runner( + workspace=workspace, + question=question, + context=context, + retrieval_evidence=retrieval_evidence, + graph_context=graph_context, + command=command, + output_mode=output_mode, + timeout_seconds=timeout_seconds, + max_context_chars=max_context_chars, + ) return answer.to_markdown() @@ -157,16 +210,10 @@ def _run_codex_host_runner_sync( ) model_name = (model or os.environ.get("RB_HOST_MODEL") or DEFAULT_CODEX_HOST_MODEL).strip() - timeout = _coerce_float( - timeout_seconds if timeout_seconds is not None else os.environ.get("RB_HOST_TIMEOUT_SECONDS"), - DEFAULT_HOST_TIMEOUT_SECONDS, - ) - max_chars = _coerce_int( - max_context_chars if max_context_chars is not None else os.environ.get("RB_HOST_MAX_CONTEXT_CHARS"), - DEFAULT_HOST_MAX_CONTEXT_CHARS, - ) + timeout = _resolve_timeout(timeout_seconds) + max_chars = _resolve_max_context_chars(max_context_chars) - prompt = _build_codex_prompt( + prompt = _build_host_prompt( workspace=workspace, question=question, context=context, @@ -191,37 +238,233 @@ def _run_codex_host_runner_sync( prompt=prompt, ) - try: - completed = subprocess.run( - cmd, - cwd=str(workspace), - text=True, - capture_output=True, - timeout=timeout if timeout > 0 else None, - check=False, - ) - except subprocess.TimeoutExpired as exc: - raise HostRunnerError( - f"Codex host runner timed out after {timeout:g}s." - ) from exc - except OSError as exc: - raise HostRunnerError( - f"Failed to run Codex host runner: {_redact_secrets(str(exc))}" - ) from exc + completed = _run_host_subprocess( + cmd, + workspace=workspace, + timeout=timeout, + label="Codex host runner", + ) + return _read_host_answer( + completed, + output_path=output_path, + label="Codex host runner", + ) + + +async def run_generic_host_runner( + *, + workspace: Path, + question: str, + context: str, + retrieval_evidence: str | None = None, + graph_context: str | None = None, + command: str | None = None, + output_mode: str | None = None, + timeout_seconds: float | None = None, + max_context_chars: int | None = None, +) -> HostRunnerAnswer: + """Answer via an arbitrary headless agent CLI configured by ``RB_HOST_COMMAND``.""" + import asyncio + + return await asyncio.to_thread( + _run_generic_host_runner_sync, + workspace=workspace, + question=question, + context=context, + retrieval_evidence=retrieval_evidence, + graph_context=graph_context, + command=command, + output_mode=output_mode, + timeout_seconds=timeout_seconds, + max_context_chars=max_context_chars, + ) + + +def normalize_generic_output_mode(value: str | None) -> str: + """Normalize ``RB_HOST_OUTPUT_MODE`` to a supported value.""" + mode = (value or "").strip().lower() + if not mode: + return DEFAULT_GENERIC_OUTPUT_MODE + if mode not in GENERIC_OUTPUT_MODES: + supported = ", ".join(sorted(GENERIC_OUTPUT_MODES)) + raise HostRunnerError( + f"Unsupported RB_HOST_OUTPUT_MODE={value!r}. Supported values: {supported}." + ) + return mode + + +def build_generic_command( + *, + template: str, + workspace: Path, + schema_path: Path, + output_path: Path, + prompt_path: Path, +) -> tuple[list[str], bool]: + """Render ``RB_HOST_COMMAND`` into an argv list. + + Returns the argv and whether the template referenced ``{prompt_file}``. When + it did not, the prompt must be delivered on stdin by the caller. + + Placeholders: ``{prompt_file}``, ``{schema_file}``, ``{output_file}``, + ``{workspace}``. + """ + template = (template or "").strip() + if not template: + raise HostRunnerError( + "RB_HOST_RUNNER=generic requires RB_HOST_COMMAND to be set, e.g. " + "RB_HOST_COMMAND='mycli exec --prompt {prompt_file}'." + ) + + uses_prompt_file = GENERIC_PROMPT_FILE_PLACEHOLDER in template + substitutions = { + GENERIC_PROMPT_FILE_PLACEHOLDER: str(prompt_path), + GENERIC_SCHEMA_FILE_PLACEHOLDER: str(schema_path), + GENERIC_OUTPUT_FILE_PLACEHOLDER: str(output_path), + GENERIC_WORKSPACE_PLACEHOLDER: str(workspace), + } + + try: + tokens = shlex.split(template) + except ValueError as exc: + raise HostRunnerError( + f"Invalid RB_HOST_COMMAND (could not parse): {exc}" + ) from exc + if not tokens: + raise HostRunnerError("RB_HOST_COMMAND parsed to an empty command.") + + argv = [_apply_placeholders(token, substitutions) for token in tokens] + return argv, uses_prompt_file + + +def _apply_placeholders(token: str, substitutions: dict[str, str]) -> str: + for placeholder, value in substitutions.items(): + token = token.replace(placeholder, value) + return token - if completed.returncode != 0: - stderr = _redact_secrets((completed.stderr or completed.stdout or "").strip()) + +def _run_generic_host_runner_sync( + *, + workspace: Path, + question: str, + context: str, + retrieval_evidence: str | None, + graph_context: str | None, + command: str | None, + output_mode: str | None, + timeout_seconds: float | None, + max_context_chars: int | None, +) -> HostRunnerAnswer: + template = command if command is not None else os.environ.get("RB_HOST_COMMAND") + mode = normalize_generic_output_mode( + output_mode if output_mode is not None else os.environ.get("RB_HOST_OUTPUT_MODE") + ) + timeout = _resolve_timeout(timeout_seconds) + max_chars = _resolve_max_context_chars(max_context_chars) + + prompt = _build_host_prompt( + workspace=workspace, + question=question, + context=context, + retrieval_evidence=retrieval_evidence, + graph_context=graph_context, + max_context_chars=max_chars, + ) + + with tempfile.TemporaryDirectory(prefix="rb-host-runner-") as tmp_dir: + tmp_path = Path(tmp_dir) + schema_path = tmp_path / "generic_host_answer.schema.json" + output_path = tmp_path / "generic_host_answer.json" + prompt_path = tmp_path / "generic_host_prompt.txt" + schema_path.write_text( + json.dumps(_host_answer_schema(), ensure_ascii=False, indent=2), + encoding="utf-8", + ) + prompt_path.write_text(prompt, encoding="utf-8") + + argv, uses_prompt_file = build_generic_command( + template=template, + workspace=workspace, + schema_path=schema_path, + output_path=output_path, + prompt_path=prompt_path, + ) + + # Verify the host executable resolves before launching, for a clear error. + executable = argv[0] + if shutil.which(executable) is None and not Path(executable).exists(): raise HostRunnerError( - "Codex host runner failed" - + (f": {stderr[:1200]}" if stderr else ".") + f"Generic host runner command not found on PATH: {executable!r}. " + "Check RB_HOST_COMMAND points at an installed CLI." ) - raw_output = "" - if output_path.is_file(): - raw_output = output_path.read_text(encoding="utf-8").strip() - if not raw_output: - raw_output = (completed.stdout or "").strip() - return parse_host_runner_answer(raw_output) + # When the template does not reference {prompt_file}, feed prompt on stdin. + stdin_text = None if uses_prompt_file else prompt + completed = _run_host_subprocess( + argv, + workspace=workspace, + timeout=timeout, + label="Generic host runner", + stdin_text=stdin_text, + ) + + expected_output = output_path if mode == "file" else None + return _read_host_answer( + completed, + output_path=expected_output, + label="Generic host runner", + ) + + +def _run_host_subprocess( + cmd: list[str], + *, + workspace: Path, + timeout: float, + label: str, + stdin_text: str | None = None, +) -> subprocess.CompletedProcess: + """Run a host-runner subprocess with shared timeout/error handling.""" + try: + completed = subprocess.run( + cmd, + cwd=str(workspace), + text=True, + input=stdin_text, + capture_output=True, + timeout=timeout if timeout > 0 else None, + check=False, + ) + except subprocess.TimeoutExpired as exc: + raise HostRunnerError(f"{label} timed out after {timeout:g}s.") from exc + except OSError as exc: + raise HostRunnerError( + f"Failed to run {label}: {_redact_secrets(str(exc))}" + ) from exc + + if completed.returncode != 0: + stderr = _redact_secrets((completed.stderr or completed.stdout or "").strip()) + raise HostRunnerError( + f"{label} failed" + (f": {stderr[:1200]}" if stderr else ".") + ) + return completed + + +def _read_host_answer( + completed: subprocess.CompletedProcess, + *, + output_path: Path | None, + label: str, +) -> HostRunnerAnswer: + """Recover the JSON answer from an output file (if any) then stdout.""" + raw_output = "" + if output_path is not None and output_path.is_file(): + raw_output = output_path.read_text(encoding="utf-8").strip() + if not raw_output: + raw_output = (completed.stdout or "").strip() + if not raw_output: + raise HostRunnerError(f"{label} returned no output.") + return parse_host_runner_answer(raw_output) def parse_host_runner_answer(raw_output: str) -> HostRunnerAnswer: @@ -229,7 +472,7 @@ def parse_host_runner_answer(raw_output: str) -> HostRunnerAnswer: payload = _parse_json_object(raw_output) answer = str(payload.get("answer") or "").strip() if not answer: - raise HostRunnerError("Codex host runner returned JSON without a non-empty answer.") + raise HostRunnerError("Host runner returned JSON without a non-empty answer.") return HostRunnerAnswer( answer=answer, sources=_coerce_string_list(payload.get("sources")), @@ -240,7 +483,7 @@ def parse_host_runner_answer(raw_output: str) -> HostRunnerAnswer: def _parse_json_object(raw_output: str) -> dict[str, Any]: text = (raw_output or "").strip() if not text: - raise HostRunnerError("Codex host runner returned no output.") + raise HostRunnerError("Host runner returned no output.") try: payload = json.loads(text) @@ -251,30 +494,30 @@ def _parse_json_object(raw_output: str) -> dict[str, Any]: payload = json.loads(fenced.group(1)) except json.JSONDecodeError as exc: raise HostRunnerError( - "Codex host runner returned invalid JSON in a fenced block." + "Host runner returned invalid JSON in a fenced block." ) from exc else: start = text.find("{") end = text.rfind("}") if start == -1 or end == -1 or end < start: raise HostRunnerError( - "Codex host runner returned non-JSON output. " + "Host runner returned non-JSON output. " f"Preview: {_redact_secrets(text[:400])}" ) try: payload = json.loads(text[start : end + 1]) except json.JSONDecodeError as exc: raise HostRunnerError( - "Codex host runner returned malformed JSON. " + "Host runner returned malformed JSON. " f"Preview: {_redact_secrets(text[:400])}" ) from exc if not isinstance(payload, dict): - raise HostRunnerError("Codex host runner JSON output must be an object.") + raise HostRunnerError("Host runner JSON output must be an object.") return payload -def _build_codex_prompt( +def _build_host_prompt( *, workspace: Path, question: str, @@ -285,7 +528,7 @@ def _build_codex_prompt( ) -> str: sections = [ ( - "You are RepoBrain's local Codex host runner for read-only codebase Q&A.\n" + "You are RepoBrain's local host runner for read-only codebase Q&A.\n" "You may inspect files in the workspace, but you must not modify files, run " "formatters, create commits, or perform network or write-side effects.\n" "Use the supplied RepoBrain context first, then verify with source files " @@ -313,6 +556,10 @@ def _build_codex_prompt( return prompt +# Backwards-compatible alias — some callers/tests may import the codex name. +_build_codex_prompt = _build_host_prompt + + def _host_answer_schema() -> dict[str, Any]: return { "type": "object", @@ -358,6 +605,26 @@ def _coerce_int(value: object, default: int) -> int: return default +def _resolve_timeout(timeout_seconds: float | None) -> float: + """Resolve the host-runner timeout from an explicit arg or env fallback.""" + return _coerce_float( + timeout_seconds + if timeout_seconds is not None + else os.environ.get("RB_HOST_TIMEOUT_SECONDS"), + DEFAULT_HOST_TIMEOUT_SECONDS, + ) + + +def _resolve_max_context_chars(max_context_chars: int | None) -> int: + """Resolve the max prompt size from an explicit arg or env fallback.""" + return _coerce_int( + max_context_chars + if max_context_chars is not None + else os.environ.get("RB_HOST_MAX_CONTEXT_CHARS"), + DEFAULT_HOST_MAX_CONTEXT_CHARS, + ) + + def _redact_secrets(text: str) -> str: redacted = text for key, value in os.environ.items(): @@ -368,3 +635,326 @@ def _redact_secrets(text: str) -> str: redacted = redacted.replace(value, "") redacted = re.sub(r"sk-[A-Za-z0-9_-]{12,}", "sk-", redacted) return redacted + + +# --------------------------------------------------------------------------- +# Free-form text generation (used by rb-refresh via HostRunnerModel) +# +# Unlike rb-ask, the refresh module swarm expects raw Markdown, not the +# answer/sources/limitations JSON envelope. These helpers run the same host CLI +# but return the model's text output verbatim. +# --------------------------------------------------------------------------- + + +def build_codex_text_command( + *, + workspace: Path, + model: str, + output_path: Path, + prompt: str, +) -> list[str]: + """Build ``codex exec`` for a read-only free-form text generation.""" + return [ + "codex", + "exec", + "--cd", + str(workspace), + "--sandbox", + "read-only", + "--ephemeral", + "--skip-git-repo-check", + "--model", + model, + "--output-last-message", + str(output_path), + prompt, + ] + + +def run_host_text_generation( + *, + runner: str, + prompt: str, + workspace: Path, + model: str | None = None, + command: str | None = None, + output_mode: str | None = None, + timeout_seconds: float | None = None, +) -> str: + """Run a host CLI for a single free-form text completion and return its text. + + This is the synchronous core shared by both the ``codex`` and ``generic`` + runners when driving ``rb-refresh`` through :class:`HostRunnerModel`. + """ + runner_name = normalize_host_runner_name(runner) + if runner_name not in SUPPORTED_HOST_RUNNERS: + supported = ", ".join(sorted(SUPPORTED_HOST_RUNNERS)) + raise HostRunnerError( + f"Unsupported host runner: {runner or ''}. " + f"Supported values: {supported}." + ) + + timeout = _resolve_timeout(timeout_seconds) + + if runner_name == "codex": + if shutil.which("codex") is None: + raise HostRunnerError( + "Codex CLI is not installed or not on PATH. Install Codex CLI and run " + "`codex login` before using RB_HOST_RUNNER=codex." + ) + model_name = ( + model or os.environ.get("RB_HOST_MODEL") or DEFAULT_CODEX_HOST_MODEL + ).strip() + with tempfile.TemporaryDirectory(prefix="rb-host-runner-") as tmp_dir: + output_path = Path(tmp_dir) / "codex_host_text.txt" + cmd = build_codex_text_command( + workspace=workspace, + model=model_name, + output_path=output_path, + prompt=prompt, + ) + completed = _run_host_subprocess( + cmd, workspace=workspace, timeout=timeout, label="Codex host runner" + ) + return _read_host_text(completed, output_path=output_path, label="Codex host runner") + + # generic + template = command if command is not None else os.environ.get("RB_HOST_COMMAND") + mode = normalize_generic_output_mode( + output_mode if output_mode is not None else os.environ.get("RB_HOST_OUTPUT_MODE") + ) + with tempfile.TemporaryDirectory(prefix="rb-host-runner-") as tmp_dir: + tmp_path = Path(tmp_dir) + schema_path = tmp_path / "generic_host_answer.schema.json" + output_path = tmp_path / "generic_host_text.txt" + prompt_path = tmp_path / "generic_host_prompt.txt" + # Schema is unused for text mode but kept so the same template works for + # both ask (JSON) and refresh (text); the CLI may ignore {schema_file}. + schema_path.write_text( + json.dumps(_host_answer_schema(), ensure_ascii=False, indent=2), + encoding="utf-8", + ) + prompt_path.write_text(prompt, encoding="utf-8") + + argv, uses_prompt_file = build_generic_command( + template=template, + workspace=workspace, + schema_path=schema_path, + output_path=output_path, + prompt_path=prompt_path, + ) + executable = argv[0] + if shutil.which(executable) is None and not Path(executable).exists(): + raise HostRunnerError( + f"Generic host runner command not found on PATH: {executable!r}. " + "Check RB_HOST_COMMAND points at an installed CLI." + ) + stdin_text = None if uses_prompt_file else prompt + completed = _run_host_subprocess( + argv, + workspace=workspace, + timeout=timeout, + label="Generic host runner", + stdin_text=stdin_text, + ) + expected_output = output_path if mode == "file" else None + return _read_host_text( + completed, output_path=expected_output, label="Generic host runner" + ) + + +def _read_host_text( + completed: subprocess.CompletedProcess, + *, + output_path: Path | None, + label: str, +) -> str: + """Recover free-form text from an output file (if any) then stdout.""" + raw_output = "" + if output_path is not None and output_path.is_file(): + raw_output = output_path.read_text(encoding="utf-8").strip() + if not raw_output: + raw_output = (completed.stdout or "").strip() + if not raw_output: + raise HostRunnerError(f"{label} returned no output.") + return raw_output + + +class HostRunnerModel: + """An Agents-SDK ``Model`` backed by a local headless CLI (no API key). + + This lets ``rb-refresh`` drive the preloaded module swarm through the user's + local Codex login or any generic CLI. It only supports **single-turn, + text-only** generation: agents that require function/tool calling or + handoffs cannot be represented faithfully by a text-only CLI and raise a + clear error so the caller can fall back. + + The SDK's runner resolves ``agent.model`` with ``isinstance(model, Model)`` + where ``Model`` is an ABC. We register this class as a *virtual* subclass + lazily (see :func:`_register_as_agents_model`) so the ask-only host-runner + path never has to import ``agents`` while refresh can still hand a + ``HostRunnerModel`` straight to ``Runner.run``. + """ + + def __init__( + self, + *, + runner: str, + workspace: Path, + model: str | None = None, + command: str | None = None, + output_mode: str | None = None, + timeout_seconds: float | None = None, + ) -> None: + _register_as_agents_model() + self._runner = normalize_host_runner_name(runner) + self._workspace = workspace + self._model = model + self._command = command + self._output_mode = output_mode + self._timeout_seconds = timeout_seconds + + async def get_response( + self, + system_instructions, + input, + model_settings, + tools, + output_schema, + handoffs, + tracing, + *, + previous_response_id=None, + conversation_id=None, + prompt=None, + **kwargs, + ): + import asyncio + + from agents.items import ModelResponse + from agents.usage import Usage + + if tools: + raise HostRunnerError( + "HostRunnerModel does not support tool calls. The " + f"'{self._runner}' host runner can only drive tool-free, " + "single-turn agents (e.g. the preloaded refresh module swarm)." + ) + if handoffs: + raise HostRunnerError( + "HostRunnerModel does not support handoffs. The " + f"'{self._runner}' host runner can only drive single-agent, " + "single-turn generation." + ) + + composed = _compose_model_prompt(system_instructions, input) + text = await asyncio.to_thread( + run_host_text_generation, + runner=self._runner, + prompt=composed, + workspace=self._workspace, + model=self._model, + command=self._command, + output_mode=self._output_mode, + timeout_seconds=self._timeout_seconds, + ) + + output_item = _build_text_output_item(text) + return ModelResponse(output=[output_item], usage=Usage(), response_id=None) + + def stream_response(self, *args, **kwargs): + raise HostRunnerError( + "HostRunnerModel does not support streaming responses. " + "Run refresh without STREAM_ENABLED when using a host runner." + ) + + # -- SDK Model helpers --------------------------------------------------- + # The runner calls these non-abstract ``Model`` helpers directly. Because + # HostRunnerModel is registered as a *virtual* subclass (not inherited) it + # does not pick up their defaults, so we mirror the SDK's no-op behavior. + + def get_retry_advice(self, request): + """No provider-specific retry guidance for a local CLI.""" + return None + + async def _cleanup_on_run_end(self, owner) -> None: + """No run-scoped resources to release for a subprocess-backed model.""" + return None + + +def _compose_model_prompt(system_instructions, input) -> str: + """Flatten SDK system instructions + input items into a single prompt.""" + parts: list[str] = [] + if system_instructions: + parts.append(str(system_instructions).strip()) + + if isinstance(input, str): + parts.append(input.strip()) + elif isinstance(input, list): + for item in input: + parts.append(_stringify_input_item(item)) + elif input is not None: + parts.append(str(input)) + + return "\n\n".join(part for part in parts if part and part.strip()) + + +def _stringify_input_item(item: Any) -> str: + """Best-effort extraction of text from an SDK input item.""" + if isinstance(item, str): + return item.strip() + if isinstance(item, dict): + content = item.get("content") + if isinstance(content, str): + return content.strip() + if isinstance(content, list): + texts = [] + for block in content: + if isinstance(block, dict): + text = block.get("text") or block.get("content") + if isinstance(text, str): + texts.append(text) + elif isinstance(block, str): + texts.append(block) + if texts: + return "\n".join(texts).strip() + # Fall back to a stable string form for unknown dict shapes. + return json.dumps(item, ensure_ascii=False, sort_keys=True) + return str(item) + + +def _build_text_output_item(text: str): + """Wrap free-form text in a ResponseOutputMessage the SDK understands.""" + from openai.types.responses import ResponseOutputMessage, ResponseOutputText + + return ResponseOutputMessage( + id="rb-host-runner", + content=[ResponseOutputText(text=text, type="output_text", annotations=[])], + role="assistant", + status="completed", + type="message", + ) + + +_registered_as_agents_model = False + + +def _register_as_agents_model() -> None: + """Register :class:`HostRunnerModel` as a virtual subclass of ``agents.Model``. + + The SDK runner resolves an agent's model with ``isinstance(model, Model)``. + Registering here (only when the SDK is importable) makes a + ``HostRunnerModel`` satisfy that check without ``HostRunnerModel`` having to + inherit from ``Model`` at import time — keeping the ask-only host-runner + path independent of the ``agents`` package. Best-effort: if the SDK is not + installed, refresh will surface a clear ImportError elsewhere. + """ + global _registered_as_agents_model + if _registered_as_agents_model: + return + try: + from agents.models.interface import Model + except ImportError: + return + Model.register(HostRunnerModel) + _registered_as_agents_model = True diff --git a/engine/repobrain_engine/hub/refresh_pipeline.py b/engine/repobrain_engine/hub/refresh_pipeline.py index bd84494ce..b92c2824a 100644 --- a/engine/repobrain_engine/hub/refresh_pipeline.py +++ b/engine/repobrain_engine/hub/refresh_pipeline.py @@ -258,26 +258,29 @@ async def refresh_pipeline(workspace: Path, quick: bool = False, failed_only: bo refresh_scan_only = bool(settings.RB_REFRESH_SCAN_ONLY) else: refresh_scan_only = scan_only_raw.strip().lower() in {"1", "true", "yes"} - model: str | None = None + model: "str | object | None" = None module_docs_changed = False + host_runner_mode = False if not refresh_scan_only: from agents import set_tracing_disabled from repobrain_engine.hub.agents import create_model - from repobrain_engine.hub.host_runner import normalize_host_runner_name + from repobrain_engine.hub.host_runner import is_host_runner_model set_tracing_disabled(True) - if ( - normalize_host_runner_name(settings.RB_HOST_RUNNER) - and not settings.OPENAI_API_KEY - and not settings.OPENAI_BASE_URL - ): - raise ValueError( - "RB_HOST_RUNNER is only supported for rb-ask. For no-key refresh, " - "run `RB_REFRESH_SCAN_ONLY=1 rb-refresh --workspace .` to build " - "local scan artifacts, or configure OPENAI_* for full LLM refresh." - ) model = create_model(settings) + # When no API key is configured, create_model returns a HostRunnerModel + # (local CLI). It can only drive tool-free, single-turn, no-handoff + # agents, so tool-using / handoff refresh stages fall back gracefully. + host_runner_mode = is_host_runner_model(model) + if host_runner_mode: + print( + f"[0/3] No API key configured; using local host runner " + f"'{settings.RB_HOST_RUNNER}' for LLM stages. Tool-using and " + "handoff stages (conventions, git insights) will use " + "deterministic fallbacks.", + file=sys.stderr, + ) rb_dir = _ensure_refresh_workspace_initialized(workspace) sha_file = rb_dir / ".last_refresh_sha" @@ -351,11 +354,20 @@ async def refresh_pipeline(workspace: Path, quick: bool = False, failed_only: bo print("[2/3] Quick mode: no changed files; preserving conventions.md.", file=sys.stderr) refresh_status.stages["conventions"] = "skipped" elif not refresh_scan_only: - from repobrain_engine.hub.agents import build_refresh_agent + from repobrain_engine.hub.agents import ( + build_refresh_agent, + build_single_turn_convention_agent, + ) prompt = _format_scan_report(report) - agent = build_refresh_agent(model or "") + # The default conventions swarm uses agent handoffs, which a host + # runner cannot drive. In host-runner mode, collapse it into a single + # tool-free, single-turn agent instead. + if host_runner_mode: + agent = build_single_turn_convention_agent(model) + else: + agent = build_refresh_agent(model or "") try: from agents import Runner except ImportError: @@ -709,25 +721,45 @@ async def _run_sub( refresh_status.stages["git_insights"] = "skipped" else: print(" → RefreshGitAgent analyzing git history...", file=sys.stderr) - try: - git_agent = build_refresh_git_agent(model, workspace) - await _run_with_retry( - Runner.run, - git_agent, - "Analyze the project's git history and write your git insights document.", - max_turns=25, - timeout=module_timeout, - context="Git agent", - ) - refresh_status.stages["git_insights"] = "success" - except Exception as exc: - print(f" ⚠ RefreshGitAgent failed: {exc}", file=sys.stderr) - _mark_stage_failure( - refresh_status, - stage="git_insights", - reason=str(exc), - partial=True, - ) + if host_runner_mode: + # The git agent relies on tools (git_log/git_diff/…), which a + # host runner cannot invoke. Write the pre-extracted git data + # deterministically instead of driving a tool-using agent. + try: + _write_host_runner_git_insights(workspace) + print( + " ✓ Wrote deterministic git insights (host-runner mode).", + file=sys.stderr, + ) + refresh_status.stages["git_insights"] = "partial" + except Exception as exc: + print(f" ⚠ Git insights fallback failed: {exc}", file=sys.stderr) + _mark_stage_failure( + refresh_status, + stage="git_insights", + reason=str(exc), + partial=True, + ) + else: + try: + git_agent = build_refresh_git_agent(model, workspace) + await _run_with_retry( + Runner.run, + git_agent, + "Analyze the project's git history and write your git insights document.", + max_turns=25, + timeout=module_timeout, + context="Git agent", + ) + refresh_status.stages["git_insights"] = "success" + except Exception as exc: + print(f" ⚠ RefreshGitAgent failed: {exc}", file=sys.stderr) + _mark_stage_failure( + refresh_status, + stage="git_insights", + reason=str(exc), + partial=True, + ) else: print("[7/8] Scan-only mode: module agents skipped.", file=sys.stderr) refresh_status.stages["module_docs"] = "skipped" @@ -1620,6 +1652,37 @@ def _write_agent_md_artifacts( out_path.write_text(md_content, encoding="utf-8") +def _write_host_runner_git_insights(workspace: Path) -> Path: + """Write a deterministic git insights doc for host-runner mode. + + The normal git stage drives a tool-using agent (git_log/git_diff/…), + which a :class:`HostRunnerModel` cannot invoke. This writes the + pre-extracted git data straight to ``.repobrain/modules/_git_insights.md`` + — the same artifact the agent would have produced — so downstream Q&A + still has git context, just without LLM narration. + + Args: + workspace: Project root directory. + + Returns: + Path to the written git insights document. + """ + from repobrain_engine.hub.scanner import extract_git_insights + + git_data = extract_git_insights(workspace) + modules_dir = workspace / ".repobrain" / "modules" + modules_dir.mkdir(parents=True, exist_ok=True) + doc_path = modules_dir / "_git_insights.md" + content = ( + "# Git Insights\n\n" + "_Generated in host-runner mode without an API key. This is the " + "pre-extracted git data; no LLM narration was applied._\n\n" + f"{git_data.strip()}\n" + ) + doc_path.write_text(content, encoding="utf-8") + return doc_path + + def _build_agent_md_fallback( module: str, group_name: str, diff --git a/engine/tests/test_ask_freshness.py b/engine/tests/test_ask_freshness.py index 8ad5f2661..b4fb9a663 100644 --- a/engine/tests/test_ask_freshness.py +++ b/engine/tests/test_ask_freshness.py @@ -133,3 +133,152 @@ def _fake_run(*args, **kwargs): "⚠ Knowledge base is 1 commit(s) behind HEAD -- consider running rb-refresh --quick.\n" ) assert answer.endswith("host answer") + + +# --------------------------------------------------------------------------- +# Auto-refresh gate: rb-ask refreshes itself instead of relying on an agent +# --------------------------------------------------------------------------- + + +class _AutoRefreshSettings: + """Minimal settings stub exposing only the auto-refresh knobs.""" + + def __init__(self, mode: str = "stale", lag: int = 20) -> None: + self.RB_ASK_AUTO_REFRESH = mode + self.RB_ASK_AUTO_REFRESH_LAG = lag + + +def _write_full_kb(tmp_path: Path, *, sha: str | None = "abc123") -> None: + """Write artifacts that make _structured_artifacts_available() true.""" + rb_dir = _write_repobrain(tmp_path, sha=sha) + (rb_dir / "map.md").write_text("api: docs", encoding="utf-8") + agents_dir = rb_dir / "agents" + agents_dir.mkdir() + (agents_dir / "api.md").write_text("agent docs", encoding="utf-8") + + +def test_auto_refresh_off_never_triggers(tmp_path: Path) -> None: + from repobrain_engine.hub.ask_pipeline import _should_auto_refresh + + # No KB at all, but mode is off → still no refresh. + assert _should_auto_refresh(tmp_path, _AutoRefreshSettings(mode="off")) is None + + +def test_auto_refresh_first_run_triggers_when_kb_missing(tmp_path: Path) -> None: + from repobrain_engine.hub.ask_pipeline import _should_auto_refresh + + reason = _should_auto_refresh(tmp_path, _AutoRefreshSettings(mode="first-only")) + assert reason == "no knowledge base found" + + +def test_auto_refresh_first_only_skips_when_kb_present( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + from repobrain_engine.hub.ask_pipeline import _should_auto_refresh + + _write_full_kb(tmp_path) + + # first-only must not trigger on drift, even if far behind HEAD. + def _fake_run(*args, **kwargs): + return subprocess.CompletedProcess(args[0], 0, stdout="999\n", stderr="") + + monkeypatch.setattr("subprocess.run", _fake_run) + + assert _should_auto_refresh(tmp_path, _AutoRefreshSettings(mode="first-only")) is None + + +def test_auto_refresh_stale_triggers_past_threshold( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + from repobrain_engine.hub.ask_pipeline import _should_auto_refresh + + _write_full_kb(tmp_path) + + def _fake_run(*args, **kwargs): + return subprocess.CompletedProcess(args[0], 0, stdout="25\n", stderr="") + + monkeypatch.setattr("subprocess.run", _fake_run) + + reason = _should_auto_refresh(tmp_path, _AutoRefreshSettings(mode="stale", lag=20)) + assert reason == "knowledge base is 25 commits behind HEAD" + + +def test_auto_refresh_stale_within_threshold_does_not_trigger( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + from repobrain_engine.hub.ask_pipeline import _should_auto_refresh + + _write_full_kb(tmp_path) + + def _fake_run(*args, **kwargs): + return subprocess.CompletedProcess(args[0], 0, stdout="5\n", stderr="") + + monkeypatch.setattr("subprocess.run", _fake_run) + + assert _should_auto_refresh(tmp_path, _AutoRefreshSettings(mode="stale", lag=20)) is None + + +@pytest.mark.asyncio +async def test_maybe_auto_refresh_invokes_refresh_pipeline( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """When the gate says stale, refresh_pipeline is awaited before answering.""" + import repobrain_engine.hub.refresh_pipeline as refresh_mod + from repobrain_engine.hub import ask_pipeline as ask_mod + + calls: list[tuple[Path, bool]] = [] + + async def _fake_refresh(workspace, quick: bool = False, **kwargs): + calls.append((workspace, quick)) + + monkeypatch.setattr(refresh_mod, "refresh_pipeline", _fake_refresh) + + # No KB → first-run trigger regardless of mode. + await ask_mod._maybe_auto_refresh(tmp_path, _AutoRefreshSettings(mode="stale")) + + assert calls == [(tmp_path, True)] # quick=True for the pre-answer refresh + + +@pytest.mark.asyncio +async def test_maybe_auto_refresh_swallows_refresh_failure( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """A failing auto-refresh must never block the answer.""" + import repobrain_engine.hub.refresh_pipeline as refresh_mod + from repobrain_engine.hub import ask_pipeline as ask_mod + + async def _boom(workspace, quick: bool = False, **kwargs): + raise RuntimeError("refresh exploded") + + monkeypatch.setattr(refresh_mod, "refresh_pipeline", _boom) + + # Should return normally despite the refresh error. + await ask_mod._maybe_auto_refresh(tmp_path, _AutoRefreshSettings(mode="first-only")) + + +@pytest.mark.asyncio +async def test_maybe_auto_refresh_is_reentrancy_guarded( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """A refresh already in progress must not recurse into another refresh.""" + import repobrain_engine.hub.refresh_pipeline as refresh_mod + from repobrain_engine.hub import ask_pipeline as ask_mod + + called = False + + async def _fake_refresh(workspace, quick: bool = False, **kwargs): + nonlocal called + called = True + + monkeypatch.setattr(refresh_mod, "refresh_pipeline", _fake_refresh) + monkeypatch.setattr(ask_mod, "_AUTO_REFRESH_IN_PROGRESS", True) + + await ask_mod._maybe_auto_refresh(tmp_path, _AutoRefreshSettings(mode="first-only")) + + assert called is False diff --git a/engine/tests/test_cli_entry.py b/engine/tests/test_cli_entry.py index ad2e65bcb..5ea69643e 100644 --- a/engine/tests/test_cli_entry.py +++ b/engine/tests/test_cli_entry.py @@ -145,3 +145,122 @@ def test_ask_main_keyboard_interrupt_exits_130( assert exc_info.value.code == 130 err = capsys.readouterr().err assert "Traceback" not in err + + +def test_split_answer_sections_extracts_sources_and_limitations() -> None: + """A host-runner-style answer splits into answer/sources/limitations.""" + from repobrain_engine import _cli_entry + + rendered = ( + "Auth lives in engine/hub/auth.py.\n\n" + "Sources:\n" + "- engine/hub/auth.py:12\n" + "- engine/hub/auth.py:44\n\n" + "Limitations:\n" + "- host-runner single-turn mode" + ) + + result = _cli_entry._split_answer_sections(rendered) + + assert result["answer"] == "Auth lives in engine/hub/auth.py." + assert result["sources"] == [ + "engine/hub/auth.py:12", + "engine/hub/auth.py:44", + ] + assert result["limitations"] == ["host-runner single-turn mode"] + + +def test_split_answer_sections_freeform_text_stays_whole() -> None: + """Free-form prose with no headers becomes answer with empty lists.""" + from repobrain_engine import _cli_entry + + rendered = "The Scanner class walks the tree and emits a report.\nNo headers here." + + result = _cli_entry._split_answer_sections(rendered) + + assert result["answer"] == rendered.strip() + assert result["sources"] == [] + assert result["limitations"] == [] + + +def test_ask_main_json_emits_structured_envelope( + monkeypatch: pytest.MonkeyPatch, + capsys: pytest.CaptureFixture[str], +) -> None: + """--json prints a parseable {answer, sources, limitations, ...} object.""" + import json + + from repobrain_engine import _cli_entry + + monkeypatch.setattr( + _cli_entry, + "_run_ask_pipeline", + lambda workspace, question: ( + "Auth lives in engine/hub/auth.py.\n\n" + "Sources:\n- engine/hub/auth.py:12\n\n" + "Limitations:\n- single-turn" + ), + ) + + _cli_entry.ask_main(["Where is auth?", "--workspace", ".", "--json"]) + + out = capsys.readouterr().out + payload = json.loads(out) + assert payload["answer"] == "Auth lives in engine/hub/auth.py." + assert payload["sources"] == ["engine/hub/auth.py:12"] + assert payload["limitations"] == ["single-turn"] + assert payload["question"] == "Where is auth?" + assert payload["workspace"].endswith(".") is False # resolved to abs path + + +def test_ask_main_json_errors_are_json_on_stderr( + monkeypatch: pytest.MonkeyPatch, + capsys: pytest.CaptureFixture[str], +) -> None: + """--json surfaces failures as a JSON error object, never mixed text.""" + import json + + from repobrain_engine import _cli_entry + + monkeypatch.delenv("DEBUG_MODE", raising=False) + monkeypatch.setattr( + _cli_entry, + "_run_ask_pipeline", + lambda workspace, question: (_ for _ in ()).throw(RuntimeError("boom")), + ) + + with pytest.raises(SystemExit) as exc_info: + _cli_entry.ask_main(["Where is auth?", "--workspace", ".", "--json"]) + + assert exc_info.value.code == 1 + captured = capsys.readouterr() + assert captured.out == "" # stdout stays clean for the caller + error = json.loads(captured.err) + assert "boom" in error["error"] + + +def test_ask_main_json_value_error_is_json( + monkeypatch: pytest.MonkeyPatch, + capsys: pytest.CaptureFixture[str], +) -> None: + """A ValueError (e.g. no LLM configured) is emitted as JSON under --json.""" + import json + + from repobrain_engine import _cli_entry + + monkeypatch.setattr( + _cli_entry, + "_run_ask_pipeline", + lambda workspace, question: (_ for _ in ()).throw( + ValueError("No LLM configured") + ), + ) + + with pytest.raises(SystemExit) as exc_info: + _cli_entry.ask_main(["Where is auth?", "--workspace", ".", "--json"]) + + assert exc_info.value.code == 1 + captured = capsys.readouterr() + assert captured.out == "" + error = json.loads(captured.err) + assert error["error"] == "No LLM configured" diff --git a/engine/tests/test_host_runner.py b/engine/tests/test_host_runner.py index cbb522963..3a9dd3140 100644 --- a/engine/tests/test_host_runner.py +++ b/engine/tests/test_host_runner.py @@ -1,6 +1,7 @@ """Tests for local host runner integration.""" from __future__ import annotations +import stat import subprocess from pathlib import Path from types import SimpleNamespace @@ -8,10 +9,19 @@ import pytest from repobrain_engine.hub.host_runner import ( + SUPPORTED_HOST_RUNNERS, HostRunnerError, + HostRunnerModel, build_codex_command, + build_generic_command, + is_host_runner_enabled, + is_host_runner_model, + normalize_generic_output_mode, parse_host_runner_answer, run_codex_host_runner, + run_generic_host_runner, + run_host_runner, + run_host_text_generation, ) @@ -126,3 +136,462 @@ def _run(*args, **kwargs): message = str(excinfo.value) assert "sk-super-secret-value" not in message assert "" in message + + +# --------------------------------------------------------------------------- +# Generic host runner (portable to any headless agent CLI) +# --------------------------------------------------------------------------- + + +def _write_fake_cli(tmp_path: Path, name: str, body: str) -> Path: + """Write an executable POSIX shell script acting as a fake host CLI.""" + script = tmp_path / name + script.write_text("#!/bin/sh\n" + body, encoding="utf-8") + script.chmod(script.stat().st_mode | stat.S_IEXEC | stat.S_IXGRP | stat.S_IXOTH) + return script + + +def test_registry_includes_codex_and_generic() -> None: + assert {"codex", "generic"} <= set(SUPPORTED_HOST_RUNNERS) + assert is_host_runner_enabled("codex") + assert is_host_runner_enabled("generic") + assert is_host_runner_enabled("GENERIC ") + assert not is_host_runner_enabled("trae") + assert not is_host_runner_enabled("") + + +def test_generic_command_substitutes_placeholders(tmp_path: Path) -> None: + schema_path = tmp_path / "schema.json" + output_path = tmp_path / "out.json" + prompt_path = tmp_path / "prompt.txt" + + argv, uses_prompt_file = build_generic_command( + template="mycli exec --prompt {prompt_file} --schema {schema_file} " + "--out {output_file} --cwd {workspace}", + workspace=tmp_path, + schema_path=schema_path, + output_path=output_path, + prompt_path=prompt_path, + ) + + assert uses_prompt_file is True + assert argv[0:2] == ["mycli", "exec"] + assert argv[argv.index("--prompt") + 1] == str(prompt_path) + assert argv[argv.index("--schema") + 1] == str(schema_path) + assert argv[argv.index("--out") + 1] == str(output_path) + assert argv[argv.index("--cwd") + 1] == str(tmp_path) + + +def test_generic_command_without_prompt_placeholder_signals_stdin(tmp_path: Path) -> None: + argv, uses_prompt_file = build_generic_command( + template="mycli exec", + workspace=tmp_path, + schema_path=tmp_path / "s.json", + output_path=tmp_path / "o.json", + prompt_path=tmp_path / "p.txt", + ) + assert argv == ["mycli", "exec"] + assert uses_prompt_file is False + + +def test_generic_command_empty_template_is_diagnostic(tmp_path: Path) -> None: + with pytest.raises(HostRunnerError, match="RB_HOST_COMMAND"): + build_generic_command( + template=" ", + workspace=tmp_path, + schema_path=tmp_path / "s.json", + output_path=tmp_path / "o.json", + prompt_path=tmp_path / "p.txt", + ) + + +def test_normalize_generic_output_mode() -> None: + assert normalize_generic_output_mode(None) == "file" + assert normalize_generic_output_mode("") == "file" + assert normalize_generic_output_mode("STDOUT") == "stdout" + with pytest.raises(HostRunnerError, match="RB_HOST_OUTPUT_MODE"): + normalize_generic_output_mode("pipe") + + +@pytest.mark.asyncio +async def test_generic_runner_file_mode_reads_output_file(tmp_path: Path) -> None: + cli = _write_fake_cli( + tmp_path, + "fakehost", + # Writes JSON to the file passed after --out. + 'while [ "$1" != "--out" ]; do shift; done\n' + 'shift\n' + 'printf \'{"answer":"from file","sources":["x.py:1"],"limitations":[]}\' > "$1"\n', + ) + + answer = await run_generic_host_runner( + workspace=tmp_path, + question="What?", + context="ctx", + command=f"{cli} --out {{output_file}} --prompt {{prompt_file}}", + output_mode="file", + timeout_seconds=30, + ) + + assert answer.answer == "from file" + assert answer.sources == ["x.py:1"] + + +@pytest.mark.asyncio +async def test_generic_runner_stdout_mode_reads_stdout(tmp_path: Path) -> None: + cli = _write_fake_cli( + tmp_path, + "fakehost_stdout", + 'printf \'{"answer":"from stdout","sources":[],"limitations":["partial"]}\'\n', + ) + + answer = await run_generic_host_runner( + workspace=tmp_path, + question="What?", + context="ctx", + command=f"{cli} --prompt {{prompt_file}}", + output_mode="stdout", + timeout_seconds=30, + ) + + assert answer.answer == "from stdout" + assert answer.limitations == ["partial"] + + +@pytest.mark.asyncio +async def test_generic_runner_feeds_prompt_on_stdin_without_placeholder(tmp_path: Path) -> None: + # No {prompt_file} in the template -> prompt should arrive on stdin. + # The fake CLI echoes back a JSON answer embedding the stdin length so we can + # assert the prompt was actually piped in. + cli = _write_fake_cli( + tmp_path, + "fakehost_stdin", + 'input=$(cat)\n' + 'case "$input" in\n' + ' *"unique-question-marker"*)\n' + ' printf \'{"answer":"got stdin","sources":[],"limitations":[]}\' ;;\n' + ' *)\n' + ' printf \'{"answer":"no stdin","sources":[],"limitations":[]}\' ;;\n' + 'esac\n', + ) + + answer = await run_generic_host_runner( + workspace=tmp_path, + question="unique-question-marker", + context="ctx", + command=f"{cli}", + output_mode="stdout", + timeout_seconds=30, + ) + + assert answer.answer == "got stdin" + + +@pytest.mark.asyncio +async def test_generic_runner_missing_command_is_diagnostic(tmp_path: Path) -> None: + with pytest.raises(HostRunnerError, match="RB_HOST_COMMAND"): + await run_generic_host_runner( + workspace=tmp_path, + question="What?", + context="ctx", + command="", + timeout_seconds=30, + ) + + +@pytest.mark.asyncio +async def test_generic_runner_missing_executable_is_diagnostic(tmp_path: Path) -> None: + with pytest.raises(HostRunnerError, match="not found on PATH"): + await run_generic_host_runner( + workspace=tmp_path, + question="What?", + context="ctx", + command="rb-nonexistent-cli-xyz --prompt {prompt_file}", + timeout_seconds=30, + ) + + +@pytest.mark.asyncio +async def test_generic_runner_failure_redacts_env_secrets(tmp_path: Path, monkeypatch) -> None: + monkeypatch.setenv("MY_API_KEY", "sk-super-secret-value") + cli = _write_fake_cli( + tmp_path, + "fakehost_fail", + 'echo "provider rejected sk-super-secret-value" 1>&2\n' + 'exit 3\n', + ) + + with pytest.raises(HostRunnerError) as excinfo: + await run_generic_host_runner( + workspace=tmp_path, + question="What?", + context="ctx", + command=f"{cli}", + output_mode="stdout", + timeout_seconds=30, + ) + + message = str(excinfo.value) + assert "sk-super-secret-value" not in message + assert "" in message + + +@pytest.mark.asyncio +async def test_generic_runner_timeout_reports_deadline(tmp_path: Path) -> None: + cli = _write_fake_cli(tmp_path, "fakehost_slow", "sleep 5\n") + + with pytest.raises(HostRunnerError, match="timed out after"): + await run_generic_host_runner( + workspace=tmp_path, + question="What?", + context="ctx", + command=f"{cli}", + output_mode="stdout", + timeout_seconds=1, + ) + + +@pytest.mark.asyncio +async def test_run_host_runner_dispatches_generic(tmp_path: Path) -> None: + cli = _write_fake_cli( + tmp_path, + "fakehost_dispatch", + 'printf \'{"answer":"dispatched","sources":[],"limitations":[]}\'\n', + ) + + markdown = await run_host_runner( + runner="generic", + workspace=tmp_path, + question="What?", + context="ctx", + command=f"{cli}", + output_mode="stdout", + timeout_seconds=30, + ) + + assert "dispatched" in markdown + + +@pytest.mark.asyncio +async def test_run_host_runner_rejects_unknown_runner(tmp_path: Path) -> None: + with pytest.raises(HostRunnerError, match="Supported values"): + await run_host_runner( + runner="trae", + workspace=tmp_path, + question="What?", + context="ctx", + ) + + +# --------------------------------------------------------------------------- +# HostRunnerModel — Agents-SDK Model adapter for rb-refresh (no API key) +# --------------------------------------------------------------------------- + + +def _settings(**overrides) -> SimpleNamespace: + """Build a minimal Settings-like object for create_model().""" + base = dict( + OPENAI_BASE_URL="", + OPENAI_API_KEY="", + OPENAI_MODEL="gpt-4o-mini", + RB_HOST_RUNNER="", + RB_HOST_MODEL="gpt-5.3-codex-spark", + RB_HOST_COMMAND="", + RB_HOST_OUTPUT_MODE="file", + RB_HOST_TIMEOUT_SECONDS=240.0, + project_root_path=Path("."), + ) + base.update(overrides) + return SimpleNamespace(**base) + + +def test_create_model_prefers_api_key_over_host_runner() -> None: + from repobrain_engine.hub.agents import create_model + + # Even with a host runner configured, a real API backend wins. + model = create_model( + _settings(OPENAI_API_KEY="sk-x", RB_HOST_RUNNER="generic", RB_HOST_COMMAND="cli") + ) + assert model == "gpt-4o-mini" + assert not is_host_runner_model(model) + + +def test_create_model_prefers_base_url_over_host_runner(monkeypatch) -> None: + from repobrain_engine.hub.agents import create_model + + monkeypatch.delenv("OPENAI_API_BASE", raising=False) + model = create_model( + _settings(OPENAI_BASE_URL="http://localhost:11434/v1", RB_HOST_RUNNER="codex") + ) + assert model == "litellm/openai/gpt-4o-mini" + assert not is_host_runner_model(model) + + +def test_create_model_falls_back_to_host_runner_without_api_key() -> None: + from repobrain_engine.hub.agents import create_model + + model = create_model( + _settings(RB_HOST_RUNNER="generic", RB_HOST_COMMAND="cli {prompt_file}") + ) + assert isinstance(model, HostRunnerModel) + assert is_host_runner_model(model) + + +def test_create_model_without_any_backend_raises() -> None: + from repobrain_engine.hub.agents import create_model + + with pytest.raises(ValueError, match="No LLM configured"): + create_model(_settings()) + + +def test_host_runner_model_is_virtual_agents_model() -> None: + from agents.models.interface import Model + + model = HostRunnerModel( + runner="generic", workspace=Path("."), command="cli {prompt_file}" + ) + # The SDK runner resolves agent.model with isinstance(model, Model). + assert isinstance(model, Model) + + +def test_is_host_runner_model_rejects_plain_strings() -> None: + assert is_host_runner_model("gpt-4o") is False + assert is_host_runner_model(None) is False + + +@pytest.mark.asyncio +async def test_host_runner_model_text_generation_via_generic_cli(tmp_path: Path) -> None: + from agents.model_settings import ModelSettings + from agents.models.interface import ModelTracing + + cli = _write_fake_cli( + tmp_path, + "faketext", + # Consume stdin, emit raw Markdown (no JSON envelope) on stdout. + 'cat > /dev/null\n' + 'printf \'# Module\\n\\nDoes things.\'\n', + ) + model = HostRunnerModel( + runner="generic", + workspace=tmp_path, + command=f"{cli}", + output_mode="stdout", + timeout_seconds=30, + ) + + response = await model.get_response( + "You are a refresh module agent.", + "Analyze the preloaded source.", + ModelSettings(), + [], # no tools + None, + [], # no handoffs + ModelTracing.DISABLED, + previous_response_id=None, + conversation_id=None, + prompt=None, + ) + + text = response.output[0].content[0].text + assert text == "# Module\n\nDoes things." + + +@pytest.mark.asyncio +async def test_host_runner_model_rejects_tools(tmp_path: Path) -> None: + from agents.model_settings import ModelSettings + from agents.models.interface import ModelTracing + + model = HostRunnerModel(runner="codex", workspace=tmp_path) + with pytest.raises(HostRunnerError, match="does not support tool calls"): + await model.get_response( + "sys", + "q", + ModelSettings(), + [object()], # non-empty tools + None, + [], + ModelTracing.DISABLED, + previous_response_id=None, + conversation_id=None, + prompt=None, + ) + + +@pytest.mark.asyncio +async def test_host_runner_model_rejects_handoffs(tmp_path: Path) -> None: + from agents.model_settings import ModelSettings + from agents.models.interface import ModelTracing + + model = HostRunnerModel(runner="codex", workspace=tmp_path) + with pytest.raises(HostRunnerError, match="does not support handoffs"): + await model.get_response( + "sys", + "q", + ModelSettings(), + [], + None, + [object()], # non-empty handoffs + ModelTracing.DISABLED, + previous_response_id=None, + conversation_id=None, + prompt=None, + ) + + +def test_host_runner_model_rejects_streaming(tmp_path: Path) -> None: + model = HostRunnerModel(runner="codex", workspace=tmp_path) + with pytest.raises(HostRunnerError, match="does not support streaming"): + model.stream_response() + + +@pytest.mark.asyncio +async def test_host_runner_model_drives_single_turn_agent_end_to_end(tmp_path: Path) -> None: + """A real Runner.run over a tool-free agent works through HostRunnerModel.""" + from agents import Agent, Runner, set_tracing_disabled + + set_tracing_disabled(True) + cli = _write_fake_cli( + tmp_path, + "faketext_e2e", + 'cat > /dev/null\n' + 'printf \'# Conventions\\n\\nPython project.\'\n', + ) + model = HostRunnerModel( + runner="generic", + workspace=tmp_path, + command=f"{cli}", + output_mode="stdout", + timeout_seconds=30, + ) + agent = Agent(name="ConvSingle", instructions="Write conventions.", model=model) + + result = await Runner.run(agent, "Analyze the scan report and write conventions.") + + assert result.final_output == "# Conventions\n\nPython project." + + +def test_run_host_text_generation_rejects_unknown_runner(tmp_path: Path) -> None: + with pytest.raises(HostRunnerError, match="Supported values"): + run_host_text_generation( + runner="trae", + prompt="hi", + workspace=tmp_path, + ) + + +def test_run_host_text_generation_generic_stdout(tmp_path: Path) -> None: + cli = _write_fake_cli( + tmp_path, + "faketext_stdout", + 'cat > /dev/null\n' + 'printf \'plain markdown, not json\'\n', + ) + text = run_host_text_generation( + runner="generic", + prompt="Summarize.", + workspace=tmp_path, + command=f"{cli}", + output_mode="stdout", + timeout_seconds=30, + ) + assert text == "plain markdown, not json"