Ralph testing

.gitignore

@@ -11,3 +11,5 @@ wheels/
 dev_scratch
 .dingllm/files.txt
 llm.md
+.llm.md
+logs/

devlogs.md

@@ -53,3 +53,93 @@ Add `--brief/-b` flag to hide intermediate tool calls/results, showing only fina
 
 ### Use Case
 Useful when using buzzllm as a Claude Code skill - prevents flooding context with intermediate steps.
+
+## 2026-02-15: Iter 1 M1 Tool Catalog
+
+Timestamp: 2026-02-15
+Goal: Add tool catalog + helper for tool subset schemas.
+Changes: Added tool catalog mapping and schema builder; added unit tests for tool subset validation.
+Commands+Results: (none)
+Decisions: Keep tool catalog as a single mapping with per-tool descriptions.
+Next: Implement subagent runner + SSE parser.
+Checkpoint: 82170a5
+
+## 2026-02-15: Iter 1 M2 Subagent Runner
+
+Timestamp: 2026-02-15
+Goal: Add subagent runner + SSE output parser.
+Changes: Added subagent runner entrypoint and SSE output_text parser; added unit test for mixed SSE events.
+Commands+Results: (none)
+Decisions: Keep SSE parsing minimal and output_text-only for subagent output.
+Next: Add call_subagent tool and main wiring.
+Checkpoint: a2db6f5
+
+## 2026-02-15: Iter 1 M3 call_subagent Tool
+
+Timestamp: 2026-02-15
+Goal: Add call_subagent tool and wire into main tool list.
+Changes: Added subagent context/config + payload builder, call_subagent execution, and integration/unit tests.
+Commands+Results: (none)
+Decisions: Validate tool_subset against catalog before spawning subprocess.
+Next: Implement openai-responses tooling updates.
+Checkpoint: 6587e1f
+
+## 2026-02-15: Iter 1 Responses + Structured Outputs
+
+Timestamp: 2026-02-15
+Goal: Add OpenAI Responses tool loop support and structured output handling.
+Changes: Added responses input/tool handling, tool-call parsing, structured output buffering, and unit/integration tests.
+Commands+Results: (none)
+Decisions: Use response_format/output_config for structured outputs and previous_response_id for responses tool turns.
+Next: Run full pytest verifier.
+Checkpoint: d9e7781
+
+## 2026-02-15: Iter 1 Pytest Run 1
+
+Timestamp: 2026-02-15
+Goal: Run full pytest verifier.
+Changes: Ran full test suite.
+Commands+Results: uv run pytest -> 5 failed (openai responses integration 400s, subagent SSE JSON decode, pythonexec docker image missing).
+Decisions: Update responses reasoning payload, fix subagent SSE fixture JSON, skip docker tests when image missing.
+Next: Apply fixes and rerun pytest.
+Checkpoint: 2ad2c37
+
+## 2026-02-15: Iter 1 Pytest Run 2 (Targeted)
+
+Timestamp: 2026-02-15
+Goal: Re-run failing tests after fixes.
+Changes: Adjusted responses reasoning usage, tool schema conversion, tool-call parsing, docker image availability check, and subagent SSE fixture.
+Commands+Results: uv run pytest (targeted) -> first run 2 failures (responses tool call), follow-up tool-call-only run passed.
+Decisions: Parse function_call events for responses and force tool_choice=required in integration test.
+Next: Run full pytest verifier.
+Checkpoint: 2ad2c37
+
+## 2026-02-15: Iter 1 Pytest Run 3 (Full)
+
+Timestamp: 2026-02-15
+Goal: Run full pytest after responses fixes.
+Changes: Ran full test suite.
+Commands+Results: uv run pytest -> 1 failed (responses output_item.done unit test missing added event).
+Decisions: Update unit test to include output_item.added before output_item.done.
+Next: Rerun full pytest.
+Checkpoint: 2ad2c37
+
+## 2026-02-15: Iter 1 Pytest Run 4 (Full)
+
+Timestamp: 2026-02-15
+Goal: Confirm full pytest passes.
+Changes: Ran full test suite after last unit test fix.
+Commands+Results: uv run pytest -> 191 passed, 3 skipped.
+Decisions: None.
+Next: Commit fixes and summarize.
+Checkpoint: 2c09470
+
+## 2026-02-15: Iter 2 Responses Reasoning Effort
+
+Timestamp: 2026-02-15
+Goal: Align responses reasoning effort contract and keep integration passing.
+Changes: Set responses reasoning effort to none when think is false; updated request-args unit test; moved responses integration model to gpt-5.1 and added response status assertion.
+Commands+Results: uv run pytest -> 3 failed (responses integration 400s, request-args expectation); uv run pytest tests/integration/test_openai_responses_integration.py::TestOpenAIResponsesIntegration::test_simple_responses_output_text -s -> 400 reasoning.effort unsupported; uv run pytest tests/integration/test_openai_responses_integration.py::TestOpenAIResponsesIntegration::test_simple_responses_output_text -s -> 400 temperature unsupported; uv run pytest tests/integration/test_openai_responses_integration.py::TestOpenAIResponsesIntegration::test_simple_responses_output_text -s -> passed; uv run pytest -> 191 passed, 3 skipped.
+Decisions: Use gpt-5.1 for responses integration to support reasoning effort none with temperature.
+Next: None.
+Checkpoint: 3b673aa

plans/API.md

@@ -0,0 +1,98 @@
+# API
+
+1. Tool: call_subagent
+   1. Signature: call_subagent(prompt: str, system_prompt: str | None = None, tool_subset: list[str] | None = None) -> str
+   2. Behavior: spawns subprocess; parses SSE; returns concatenated output_text.
+   3. Error behavior: returns "Error: <message>" string on validation or subprocess failure.
+2. Subprocess input contract (stdin JSON)
+   1. Fields: model, provider, url, api_key_name, prompt, system_prompt, tool_subset, think, temperature, max_tokens.
+   2. Example payload:
+      {
+        "model": "<parent model>",
+        "provider": "openai-chat|anthropic|vertexai-anthropic",
+        "url": "<parent url>",
+        "api_key_name": "<env var>",
+        "prompt": "...",
+        "system_prompt": "...",
+        "tool_subset": ["search_web", "scrape_webpage"],
+        "think": false,
+        "temperature": 0.8,
+        "max_tokens": 8192
+      }
+3. SSE parsing contract
+   1. Only consume events with "event: output_text".
+   2. Ignore tool_call, tool_result, reasoning_content, block_end, response_end.
+
+Available tool names
+1. search_web
+2. scrape_webpage
+3. bash_find
+4. bash_ripgrep
+5. bash_read
+6. python_execute
+
+## OpenAI Responses API Contract
+
+1. Request args (make_openai_responses_request_args)
+   1. data.model = opts.model
+   2. data.input = [
+      {"role": "user", "content": [{"type": "input_text", "text": prompt}]}
+      ]
+   3. data.instructions = system_prompt when provided
+   4. data.stream = True, data.store = False
+   5. data.temperature = opts.temperature
+   6. data.max_output_tokens = opts.max_tokens when provided
+   7. data.reasoning = {"effort": "high", "summary": "detailed"} if opts.think else {"effort": "none"}
+   8. if opts.tools: data.tools = opts.tools and data.tool_choice = "auto"
+
+2. Streaming event mapping (handle_openai_responses_stream_response)
+   1. response.created -> StreamResponse(type=response_start, id=response.id)
+   2. response.output_text.delta -> StreamResponse(type=output_text, delta)
+   3. response.reasoning_summary_text.delta -> StreamResponse(type=reasoning_content, delta)
+   4. response.output_item.added (item.type=tool_call) -> create ToolCall(id=item.id, name=item.name)
+   5. response.tool_call_arguments.delta -> append ToolCall.arguments; emit StreamResponse(type=tool_call, delta)
+   6. response.output_item.done (item.type=tool_call) -> set ToolCall.arguments from item.arguments if present
+   7. response.completed -> StreamResponse(type=block_end)
+
+3. Tool response continuation (tool_call_response_to_openai_responses_messages)
+   1. request_args.data["previous_response_id"] = last_openai_response_id
+   2. request_args.data["input"] = [
+      {"type": "tool_result", "tool_call_id": tc.id, "content": str(tc.result)}
+      ] for each tool call
+   3. tool_result items replace prior input for the follow-up request (no message history replay).
+
+4. invoke_llm routing contract
+   1. If request_args.data contains "messages", use that list and write back to "messages".
+   2. Else if request_args.data contains "input", use that list and write back to "input".
+
+## Structured Outputs API Contract
+
+1. LLMOptions additions (internal)
+   1. output_mode: "json_schema" | "json_object" | None
+   2. output_schema: dict | None
+2. OpenAI chat request args
+   1. response_format json_schema:
+      {"type":"json_schema","json_schema":{"name":"<name>","schema":<schema>,"strict":true}}
+   2. response_format json_object:
+      {"type":"json_object"}
+3. Anthropic/Vertex request args
+   1. output_config: {"format":{"type":"json_schema","schema":<schema>}}
+4. Output to stdout
+   1. Non-SSE prints JSON once at end, then DONE marker.
+   2. SSE emits one output_structured event with JSON string.
+
+Non-SSE example
+```
+{"capital":"Paris","country":"France"}
+
+=== [ DONE ] ===
+```
+
+SSE example
+```
+event: output_structured
+data: {"id":"","delta":"{\"capital\":\"Paris\",\"country\":\"France\"}","type":"output_structured"}
+
+event: response_end
+data: {"id":"","delta":"","type":"response_end"}
+```

plans/ARCH.md

@@ -0,0 +1,58 @@
+# Architecture
+
+1. New modules
+   1. src/buzzllm/subagent.py: call_subagent tool + SSE parsing.
+   2. src/buzzllm/subagent_runner.py: subprocess entrypoint; reads JSON; runs invoke_llm SSE.
+   3. src/buzzllm/tools/catalog.py: tool name to callable/desc mapping.
+2. Existing modules touched
+   1. src/buzzllm/tools/utils.py: schema builder helpers for tool subsets.
+   2. src/buzzllm/main.py: include call_subagent in tool list when tools enabled.
+
+```mermaid
+graph TD
+  A[Main invoke_llm] --> B[call_subagent tool]
+  B --> C[Subprocess: subagent_runner]
+  C --> D[invoke_llm SSE]
+  D --> E[SSE stdout]
+  E --> B
+  C --> F[Tool registry (subset only)]
+```
+
+Acceptance
+1. Subprocess has isolated TOOL_CALLS and tool registry.
+2. Main agent receives only subagent output_text.
+
+## OpenAI Responses Tooling
+
+1. Existing modules touched
+   1. src/buzzllm/llm.py: make_openai_responses_request_args, handle_openai_responses_stream_response, tool_call_response_to_openai_responses_messages, invoke_llm input/messages routing.
+   2. tests/unit/test_llm_request_args.py, tests/unit/test_llm_stream_handlers.py, tests/unit/test_llm_tool_messages.py.
+   3. tests/integration/test_openai_responses_integration.py (new).
+2. In-memory state
+   1. last_openai_response_id (string) stored during streaming for previous_response_id.
+   2. input list buffer for tool_result continuation requests.
+3. No new modules; reuse existing ToolCall structure and TOOL_CALLS registry.
+
+```mermaid
+graph TD
+  A[invoke_llm] --> B[make_openai_responses_request_args]
+  B --> C[OpenAI Responses SSE]
+  C --> D[handle_openai_responses_stream_response]
+  D --> E[TOOL_CALLS populated]
+  E --> F[run_tools]
+  F --> G[tool_call_response_to_openai_responses_messages]
+  G --> B
+```
+
+Acceptance
+1. Responses tool loop uses previous_response_id without affecting chat completions.
+2. invoke_llm supports messages and input modes without branching elsewhere.
+
+## Structured Outputs Architecture
+
+1. Existing modules touched
+   1. src/buzzllm/llm.py: LLMOptions structured fields, request args (openai-chat, anthropic, vertex), invoke_llm buffering, print_to_stdout output_structured.
+2. In-memory state
+   1. structured_output_buffer (string) used only when structured output is enabled.
+3. StreamResponse type
+   1. Add output_structured to StreamResponse.type for SSE.

plans/CHECKLIST.md

@@ -0,0 +1,23 @@
+# Definition of Done
+
+1. call_subagent tool available when tools enabled.
+2. Subagent tool_subset allowlist enforced.
+3. SSE parser returns only output_text.
+4. uv run pytest passes.
+5. devlogs.md updated for final iteration.
+6. No new dependencies added.
+
+## OpenAI Responses Definition of Done
+
+1. make_openai_responses_request_args builds input list and tools when present.
+2. handle_openai_responses_stream_response emits tool_call + output_text events.
+3. Tool results are sent with previous_response_id and tool_result input items.
+4. uv run pytest passes (integration skipped without OPENAI_API_KEY).
+5. devlogs.md updated for final iteration.
+
+## Structured Outputs Definition of Done
+
+1. OpenAI chat uses response_format when output_mode is set.
+2. Anthropic/Vertex uses output_config.format when output_mode is set.
+3. Non-SSE prints JSON once; SSE emits output_structured.
+4. uv run pytest passes.

plans/DATA.md

@@ -0,0 +1,33 @@
+# Data
+
+1. Storage changes: none.
+2. Migrations: none.
+3. In-memory data:
+   1. SSE output_text buffer in src/buzzllm/subagent.py.
+   2. tool_subset list (validated against catalog).
+
+Acceptance
+1. No new files under data/ or migrations/.
+2. No persistent state introduced.
+
+## OpenAI Responses Data
+
+1. Storage changes: none.
+2. Migrations: none.
+3. In-memory data:
+   1. last_openai_response_id (string) for previous_response_id.
+   2. request_args.data["input"] list reused between tool turns.
+
+Acceptance
+1. No new persistence is introduced for responses.
+2. previous_response_id is only stored in-memory per invocation.
+
+## Structured Outputs Data
+
+1. Storage changes: none.
+2. Migrations: none.
+3. In-memory data:
+   1. structured_output_buffer (string) for output_text accumulation.
+
+Acceptance
+1. No new persistence is introduced for structured outputs.

plans/DEVLOG_CONTRACT.md

@@ -0,0 +1,14 @@
+# Devlog Contract
+
+1. Update devlogs.md at least once per iteration and after each verifier run.
+2. Use this template:
+   1. Timestamp:
+   2. Goal:
+   3. Changes:
+   4. Commands+Results:
+   5. Decisions:
+   6. Next:
+   7. Checkpoint: <commit hash>
+3. Checkpoint hash must be updated after each milestone.
+
+4. Applies to all plan sections, including OpenAI Responses Tooling milestones.

plans/FEEDBACK.md

@@ -0,0 +1 @@
+# Feedback