Skip to content

fix: unwrap JSON/tag-wrapped model output before it reaches the user - #16

Merged
Kuberwastaken merged 2 commits into
mainfrom
fix/unwrap-structured-model-output
Jul 24, 2026
Merged

fix: unwrap JSON/tag-wrapped model output before it reaches the user#16
Kuberwastaken merged 2 commits into
mainfrom
fix/unwrap-structured-model-output

Conversation

@Kuberwastaken

Copy link
Copy Markdown
Owner

Problem

Fixes #14. The on-device model occasionally returns its answer wrapped in a JSON object or <cleaned_text> tags despite the system prompt telling it not to, and normalizeCommandOutput had no handling for either — so the raw scaffolding was pasted verbatim. It's intermittent (the model usually returns bare text) and reproduces purely at the string layer, independent of the model.

normalizeCommandOutput strips model-invented wrappers from an allowlist (response, result, output, …) and has no JSON handling at all. The wrappers in the issue aren't on that list. Running the reporter's exact samples through the current pipeline on macOS 26.5.2:

model output before
<cleaned_text>…</cleaned_text> pasted verbatim, tags and all
{"cleaned_text": "…"} pasted verbatim
```json\n{"cleaned_text": "…"}\n``` pasted verbatim

Change

Unwrap the known wrappers in normalizeCommandOutput, before the existing tag loop:

  • JSON wrapper object, bare or inside a ```json fence, collapses to its text value (cleaned_text / clean_text / text / … keys).
  • <cleaned_text> / <clean_text> tags join the existing tag allowlist.

Two guards keep genuine dictation safe:

  1. A JSON object is unwrapped only when every key is a known answer-wrapper key, so a real dictated object like {"name": "Ada"} passes through untouched.
  2. A code fence is stripped only when it wraps such an object, so a code block the user actually dictated (e.g. via a wake command) keeps its fence.

normalizeCommandOutput is the single choke point for every model-output path (dictation cleanup, selection edit, transform, wake command), so this covers all of them at once.

After:

in out
<cleaned_text>\nWhat's up, my fellow humans?\n</cleaned_text> What's up, my fellow humans?
{"cleaned_text": "Testing, testing."} Testing, testing.
```json\n{"text": "- YOLO."}\n``` - YOLO.
{"name": "Ada", "text": "hello"} (unchanged)
```swift\nprint("hi")\n``` (unchanged)

Tests

New Tests/StructuredOutputUnwrapTests.swift, registered in the runner and the Makefile, covering all three reported variants plus the must-not-touch cases (genuine data object, dictated code block, ordinary prose with a stray brace). make test passes on macOS 26.5.2 (Apple silicon).

Relationship to #15

Complementary. #15 rejects transcript-dropping output and, as a side effect, makes the fenced-JSON variant of #14 fall back to basic cleanup; the bare-JSON and <cleaned_text>-tag variants still leak until this unwrap lands. They compose cleanly — normalizeCommandOutput runs before validate, so with both in, the fenced case unwraps to the smart result and passes validation instead of falling back. Only trivial overlap (the Makefile/runner test-registration line).

Note

The JSON key list includes generic keys like text, so a wake command where the user genuinely dictates a request that yields {"text": "…"} would also unwrap. Judged far rarer than the bug; easy to scope down to cleaned_text/clean_text only if preferred.

The on-device model sometimes returns its answer wrapped in a JSON object
({"cleaned_text": "…"} / {"text": "…"}), optionally inside a ```json fence,
or in <cleaned_text> tags, despite the instructions. None of these were
peeled, so the raw scaffolding was pasted verbatim (issue #14).

normalizeCommandOutput now unwraps a single-purpose JSON wrapper object (bare
or fenced) and strips <cleaned_text>/<clean_text> tags. Guards keep genuine
dictated content safe: a JSON object is only unwrapped when every key is a
known answer-wrapper key, and a code fence is removed only when it wraps such
an object, so a code block the user actually dictated survives.

Tests/StructuredOutputUnwrapTests.swift covers all three reported variants
plus the must-not-touch cases; registered in the runner and Makefile.
…red-model-output

# Conflicts:
#	Makefile
#	Tests/AppContextServiceTests.swift
@Kuberwastaken
Kuberwastaken merged commit 5a9136b into main Jul 24, 2026
1 check passed
@Kuberwastaken
Kuberwastaken deleted the fix/unwrap-structured-model-output branch July 24, 2026 08:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Output is including JSON and tags like "cleaned_text".

1 participant