ACP structured output validates concatenated tool-turn text instead of final response

[DavidBakerEffendi]

Affected surface

ACP session lifecycle or streaming

What did Anvil do?

When an ACP session/prompt includes an Anvil structured-output request and the model uses tools before its final answer, Anvil can validate the concatenation of every assistant text fragment as though it were one JSON document.

In a production SlopCop scan, both the configured synthesis model and its fallback emitted individually valid objects before tool calls and then emitted the final report object. Anvil concatenated them into this shape:

{"markdownReport":"Progress update before tool use."}{"markdownReport":"Final report."}

The resulting anvil.structuredOutput metadata was a validation error:

response is not valid JSON: trailing characters at line 1 column 159

The fallback model failed identically at column 143. Both responses ended normally with stopReason=end_turn; there was no refusal, content filter, context-length failure, or provider execution error.

This exhausted SlopCop's synthesis fallback chain and forced a no-score partial report even though all seven specialist lanes had completed successfully.

The failure mechanism is still present on current Anvil master:

• src/tool_loop.rs appends text emitted alongside tool calls to full_response, then appends the final assistant text to the same string.
• src/turn_runner.rs passes that aggregate response_text to structured_output::validate_response as one JSON document.
• The same structured-output request is sent on each model turn in the tool loop.

What should it have done?

The ACP structured-output result should validate the final structured assistant completion, not a transcript-style concatenation of intermediate tool-turn preambles and the final answer.

Intermediate text may remain available for streaming, transcript persistence, and replay, but it should not contaminate the single JSON value returned in _meta.anvil.structuredOutput.

Reproduction

Starting state:

• Start an ACP session with at least one read-only tool available.
• Send session/prompt with _meta.anvil.structuredOutput using a minimal strict schema such as:

{
  "schemaName": "ExampleReport",
  "schema": {
    "type": "object",
    "additionalProperties": false,
    "properties": {
      "markdownReport": { "type": "string" }
    },
    "required": ["markdownReport"]
  },
  "allowCoercion": false
}

Prompt or ACP request:

• Ask the agent to inspect one file with a tool and then return the required report object.

Model/tool-loop sequence:

1. The model emits {"markdownReport":"I am inspecting the requested file."} together with a tool call.
2. The tool completes.
3. The model emits {"markdownReport":"# Final report\n..."} as its final text response.

Observed result:

• The streamed/persisted transcript contains both pieces, which is expected.
• _meta.anvil.structuredOutput is validation_error because validation receives the two objects concatenated together.

Public production example:

• SlopCop case: https://slopcop.brokk.ai/case/4ace606f-c190-4306-815a-6c022f50ac7b
• SlopCop status: COMPLETED_PARTIAL
• Lane coverage: 7 completed, 0 failed, 0 inconclusive
• Failure category: MODEL_FALLBACKS_EXHAUSTED / non-JSON structured output

Session and environment

• Anvil: v0.25.0, commit 6967a9e5bb2437ec65f218ed36a59a4fb2184029
• ACP client: SlopCop 0.41, commit 3b2f01877eacd39cf7fabd76c4751032b9ab494c
• Provider/models: OpenRouter; openai/gpt-5.6-terra, then openai/gpt-5.5
• Permission mode: read-only
• Runtime: Linux container on Railway
• Structured-output schema: one required string field, markdownReport, with additionalProperties: false

The relevant aggregation/validation behavior was also verified against Anvil master at f985810a49e3c8feb6851a9c98375c3784ef8aac; upgrading alone does not currently remove the defect.

Suggested implementation boundary

Keep two distinct values:

1. The complete visible/transcript response, including text emitted before tool calls.
2. The final assistant response selected for structured-output validation.

Validate only the second value. Do not recover this by blindly parsing the last JSON-looking substring from the aggregate; the structured candidate should come from the explicit final model turn.

Acceptance criteria

• A tool-bearing ACP prompt may emit intermediate assistant text without corrupting the final structured-output result.
• _meta.anvil.structuredOutput validates only the explicit final assistant completion.
• Streaming, transcript persistence, replay events, and visible agent_response retain intermediate text according to their existing contracts.
• A regression test covers: structured text plus tool call on turn 1, tool result, valid structured final text on turn 2.
• A second test confirms that malformed final structured text still fails validation even if an earlier intermediate object was valid.
• Existing single-turn structured-output and coercion behavior remains unchanged.

Prevalence and impact

This specific SlopCop failure was rare in the sampled production window (1 synthesis_unavailable case among 128 report-bearing terminal scans over 30 days), but the mechanism is deterministic whenever a structured-output tool loop emits non-empty assistant text on more than one turn. The impact is loss of an otherwise valid final report and unnecessary model fallback cost.