fix: unwrap JSON/tag-wrapped model output before it reaches the user - #16
Merged
Conversation
The on-device model sometimes returns its answer wrapped in a JSON object
({"cleaned_text": "…"} / {"text": "…"}), optionally inside a ```json fence,
or in <cleaned_text> tags, despite the instructions. None of these were
peeled, so the raw scaffolding was pasted verbatim (issue #14).
normalizeCommandOutput now unwraps a single-purpose JSON wrapper object (bare
or fenced) and strips <cleaned_text>/<clean_text> tags. Guards keep genuine
dictated content safe: a JSON object is only unwrapped when every key is a
known answer-wrapper key, and a code fence is removed only when it wraps such
an object, so a code block the user actually dictated survives.
Tests/StructuredOutputUnwrapTests.swift covers all three reported variants
plus the must-not-touch cases; registered in the runner and Makefile.
…red-model-output # Conflicts: # Makefile # Tests/AppContextServiceTests.swift
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Fixes #14. The on-device model occasionally returns its answer wrapped in a JSON object or
<cleaned_text>tags despite the system prompt telling it not to, andnormalizeCommandOutputhad no handling for either — so the raw scaffolding was pasted verbatim. It's intermittent (the model usually returns bare text) and reproduces purely at the string layer, independent of the model.normalizeCommandOutputstrips model-invented wrappers from an allowlist (response,result,output, …) and has no JSON handling at all. The wrappers in the issue aren't on that list. Running the reporter's exact samples through the current pipeline on macOS 26.5.2:<cleaned_text>…</cleaned_text>{"cleaned_text": "…"}```json\n{"cleaned_text": "…"}\n```Change
Unwrap the known wrappers in
normalizeCommandOutput, before the existing tag loop:```jsonfence, collapses to its text value (cleaned_text/clean_text/text/ … keys).<cleaned_text>/<clean_text>tags join the existing tag allowlist.Two guards keep genuine dictation safe:
{"name": "Ada"}passes through untouched.normalizeCommandOutputis the single choke point for every model-output path (dictation cleanup, selection edit, transform, wake command), so this covers all of them at once.After:
<cleaned_text>\nWhat's up, my fellow humans?\n</cleaned_text>What's up, my fellow humans?{"cleaned_text": "Testing, testing."}Testing, testing.```json\n{"text": "- YOLO."}\n```- YOLO.{"name": "Ada", "text": "hello"}```swift\nprint("hi")\n```Tests
New
Tests/StructuredOutputUnwrapTests.swift, registered in the runner and theMakefile, covering all three reported variants plus the must-not-touch cases (genuine data object, dictated code block, ordinary prose with a stray brace).make testpasses on macOS 26.5.2 (Apple silicon).Relationship to #15
Complementary. #15 rejects transcript-dropping output and, as a side effect, makes the fenced-JSON variant of #14 fall back to basic cleanup; the bare-JSON and
<cleaned_text>-tag variants still leak until this unwrap lands. They compose cleanly —normalizeCommandOutputruns beforevalidate, so with both in, the fenced case unwraps to the smart result and passes validation instead of falling back. Only trivial overlap (the Makefile/runner test-registration line).Note
The JSON key list includes generic keys like
text, so a wake command where the user genuinely dictates a request that yields{"text": "…"}would also unwrap. Judged far rarer than the bug; easy to scope down tocleaned_text/clean_textonly if preferred.