Statblock generation guidance: schema descriptions + system prompt - #26
Conversation
Guidance currently lives in one 8-line prose blob in the user message while the schema channel is switched off by a stale premise. Empirically confirmed that OpenAI Structured Outputs accepts `description` in strict mode, so field-local invariants can travel with the fields the validators actually check. Co-authored-by: Cursor <cursoragent@cursor.com>
Policy resolved to gpt-5.4-nano while the dev env override ran a different model entirely, so prompt tuning validated in dev would not transfer. Both now agree on gpt-5.4-mini. Co-authored-by: Cursor <cursoragent@cursor.com>
Structured Outputs accepts `description` in strict mode, so the invariants the domain validators enforce can travel with the fields themselves instead of living only in prose the model reads paragraphs earlier. Describes the 14 highest-density validator fields, led by the usage-kind field matrix. Titles stay stripped: Pydantic generates them from class names, duplicating the $defs key for nothing. Strict artifact grows ~560 tokens. Also repins the model-policy test to gpt-5.4-mini alongside the policy change. Co-authored-by: Cursor <cursoragent@cursor.com>
The user message carried persona, envelope rules, and conventions mixed in with the creature description, and covered maybe three of the 43 domain validators. Conventions now live in a stable system message organized around the validator clusters that actually fire: usage-kind field exclusivity, cardinality, derived math, reference discipline, and attack shape. Names automation_support explicitly in the escape hatch. The old wording said "manual automation support", leaving the model to guess between UsageKind.manual and AutomationSupport.manual; it guessed wrong in live dogfood. Intent and context lines are omitted when unset rather than emitting filler the model can parrot back. Provider protocol carries system separately; strict mode already enforces what the old hardcoded system line asked for. Co-authored-by: Cursor <cursoragent@cursor.com>
Enum-typed fields compile to a bare $ref, and OpenAI rejects the entire request
with 400 "$ref cannot have keywords {'description'}". Four descriptions landed
there, so every generation call failed. Offline snapshot tests could not see it:
the schema serialized perfectly and only the API objected.
Moves that guidance onto the $defs enum targets, where it is still read and
still single-sourced. Adds a compiler guard so any future $ref sibling fails
closed at compile time, and an opt-in live gate that asserts the published
artifact is actually accepted.
Also adds scripts/statblock_prompt_eval.py, which measures error-severity
validator codes across four schema/prompt ablation arms so the guidance can be
judged by evidence instead of vibes.
Co-authored-by: Cursor <cursoragent@cursor.com>
WORKED EXAMPLES block (usage shapes, multiattack mechanic selection, derivation arithmetic with explicit_override escape hatch) gated behind build_system_prompt(include_examples=True) and shipped by default. Section/activation pairs and spell-group usage rules ship unconditionally. PROMPT_VERSION -> v3; harness gains a both+ex arm with 'both' kept as the no-examples control. A/B (both vs both+ex, 2 fixtures x 5 trials, gpt-5.4-mini): clean rate unchanged at 2/10; mean error codes/trial 4.0 -> 3.0; derivation mismatches 10 -> 6; SECTION_ACTIVATION_INCOHERENT 5 -> 1. USAGE_FIELDS_INCOHERENT and SPELL_GROUP_SLOTS_INCOHERENT stayed pinned at 5/10 - the prompt channel does not move those two clusters. Co-authored-by: Cursor <cursoragent@cursor.com>
The description previously said only when the field must be null and never said it is required for kind 'recharge'; text now mirrors the validator. Regenerated schema artifacts, OpenAPI, TS client, and API fixtures in the project env (pydantic 2.10.6). Note: the runbook's isolated export lane pins pydantic 2.7.4 and now fails the staleness gate closed - flagged for follow-up. Measured on the pinned cell before commit (thornvault-regent x10, gpt-5.4-mini, prose + worked examples + fixed description all live): USAGE_FIELDS_INCOHERENT stayed 10/10. The schema-description root-cause theory is falsified; payload dumps show the model writes "Recharge 5-6" in rules_text yet emits recharge_range: null under structured outputs. Kept for textual correctness, not metric movement. Harness gains --dump-definitions for payload-level post-mortems. Co-authored-by: Cursor <cursoragent@cursor.com>
Model probe on the pinned hard fixture (thornvault-regent x10, both+ex): gpt-5.6-luna cut error-severity incidences ~71% vs gpt-5.4-mini (5.6 -> 1.6 per trial); derivation, reference-cycle, legendary-resource, multiattack-element, and recharge-null clusters all collapsed to zero. The one cross-model survivor was spell_slots usage placement, so the system prompt now states it explicitly: a spellcasting element's own usage is never spell_slots; spell_slots appears only on leveled prepared/known groups. PROMPT_VERSION -> v4. Confirmation run on the shipping config (thornvault x5): 4/5 clean, sole residual UNKNOWN_PHASE_REFERENCE 1/5 (phases referenced without declaration - next cluster to tune). MODEL_POLICY gains statblock_frontier -> gpt-5.6-luna for structured_generation; .env.development override follows (gitignored). Co-authored-by: Cursor <cursoragent@cursor.com>
Drakosfire
left a comment
There was a problem hiding this comment.
Review verdict: changes required before merging the base of the stack.
The system/user prompt split, retained schema descriptions, live strict-schema acceptance test, and fail-closed $ref sibling guard are directionally sound. The stack relationship with #27 is also correct.
Two contract problems remain:
- The schema and system channels contradict each other for
Usage.kind == "manual". The field-localresource_keydescription says it must be null for every non-resourcekind, while the enum guidance, system prompt, and validator deliberately permitresource_keyformanual. Since this PR's purpose is to align those channels, the contradiction should be resolved and covered by a consistency assertion. - The published generation fixtures are stale. Runtime provenance is
statblock-generation-prompt-v4, but bothcandidate-response.jsonandrevise-replay-response.jsonstill publish v3. The exporter constructs the candidate throughGenerationServiceV1, so a true regeneration would carry v4. Regenerate both fixtures and add a test tying fixture provenance toPROMPT_VERSION; the existing fixture tests only prove model validity/coherence, not freshness.
Also update the PR description before merge: it says the prompt moved to v2 and structured_generation ships on gpt-5.4-mini, while the head actually uses prompt v4 and maps the policy to gpt-5.6-luna. That is audit/provenance drift even if the final model choice is intentional.
The failing Actions run is still the known unrelated import-time OpenAI() collection failure.
| pattern=r"^[a-z][a-z0-9_]*$", | ||
| description=( | ||
| "Required for kind 'resource' and must name a declared resources[].key. " | ||
| "Must be null for every other kind." |
There was a problem hiding this comment.
Blocking — this field-local instruction contradicts the actual manual contract. UsageKind says manual may carry uses/resource_key, the system prompt repeats that, and _validate_usage intentionally leaves both optional for manual. “Must be null for every other kind” tells the model the opposite in the schema channel. Please enumerate the kinds that actually forbid it and explicitly preserve manual as optional; add a test that the schema and prompt matrices agree.
| "model": "fixture-model", | ||
| "output_tokens": null, | ||
| "prompt_version": "statblock-generation-prompt-v1", | ||
| "prompt_version": "statblock-generation-prompt-v3", |
There was a problem hiding this comment.
Blocking — published provenance is stale. PROMPT_VERSION at this head is v4, and export_fixture_pack() creates this candidate through the live GenerationServiceV1, so a regenerated fixture would say v4. revise-replay-response.json is stale in the same way. Regenerate both and add a fixture assertion against PROMPT_VERSION so future prompt bumps cannot silently leave consumer examples behind.
Review found the field-local description contradicted the enum doc, the system prompt, and the validator: Usage.resource_key said "must be null for every other kind" while manual usage legitimately allows it (validator checks nothing on manual beyond recharge_range). The description now matches: required for resource, optional for manual, null otherwise. Also refreshes published fixture provenance (candidate + revise fixtures were captured at prompt v3 while runtime ships v4) and ties fixture tests to PROMPT_VERSION so this drift fails loudly on the next bump. Regenerated: schema artifacts, OpenAPI, TS client, candidate/revise API fixtures. Lane: 290 passed; live artifact acceptance: 3 passed. Co-authored-by: Cursor <cursoragent@cursor.com>
|
Review addressed in
Lane: 290 passed; live artifact acceptance: 3 passed. |
Drakosfire
left a comment
There was a problem hiding this comment.
Stack Review 2 — no blockers
Both findings from Stack Review 1 are resolved correctly.
Usage.resource_keyguidance now agrees across the field description,UsageKinddocumentation, the system prompt, and validator semantics: required forresource, optional formanual, null otherwise.- Candidate and revision fixtures now carry prompt v4 provenance, their schema fingerprints were regenerated with the description change, and fixture tests bind the checked-in receipt to
PROMPT_VERSIONso future drift fails loudly. - The PR body now describes the actual shipping prompt/model configuration.
I found no new correctness or stack-contract issue in 10cfddf. PR 26 is ready to merge from code review. The failing repository workflow remains the known unrelated import-time OpenAI() baseline.
Why
Statblock generation guidance lived in one 8-line prose blob in the user message, covering maybe 3 of the 43 domain validators. The schema channel — the thing the model reads at the moment it writes each field — was switched off by a stale premise. Post-
b645019, domain-invalid output is no longer a terminal failure; it is GM hand-editing in the Workbench, so every avoided validator error is avoided toil.Plan:
Docs/Plans/HANDOFF-pr26-statblock-generation-prompt-and-schema-guidance.md.What changed
descriptionis no longer stripped from the OpenAI strict artifact; the highest-density validator fields now carry guidance, led by the usage-kind field matrix.titlestays stripped (Pydantic generates it from the class name for nothing). A compiler guard fails closed on$refsiblings (OpenAI rejects them; caught only by the live lane).build_system_prompt(edition)organized around the validator clusters that actually fire: cardinality, usage-kind exclusivity, derived math, reference discipline, attack shape, section/activation pairs, spell-group rules. The user message is pure task data.PROMPT_VERSION→ v4.WORKED_EXAMPLESblock (usage shapes, multiattack, legendary actions) appended to the system prompt;both+exablation arm measures it.structured_generation→statblock_frontier(gpt-5.6-luna). The falsification series showed mini's residual failures were capability-bound: Luna cut error incidences −71% on the hard fixture (5.6 → 1.6 codes/trial).scripts/statblock_prompt_eval.py— five ablation arms, error-severity validator codes as the metric,--dump-definitionsfor payload post-mortems.prompt_version == PROMPT_VERSIONso provenance drift fails loudly.Evidence
Description support in strict mode was verified empirically (live calls), not assumed. Iterated falsification on
thornvault-regent(legendary + spellcasting hard fixture): mini saturated at 0/10 clean across six guidance configurations (USAGE_FIELDS + SPELL_GROUP_SLOTS pinned; payload dumps showed prose/typed mismatches no text channel moved). Luna + the spell-slots placement line: 4/5 clean, 0.2 codes/trial (from 0/10, 5.6 on mini at session start). Sole residual:UNKNOWN_PHASE_REFERENCE1/5.Live gate:
test_openai_accepts_the_published_strict_artifactpasses against the regenerated artifact.Test plan
uv run pytest tests/statblocks_v1/ -q(290 passed)OpenAI()initialization baseline (pre-existing on main)main