From 4561d213807fd02b3343709a4c8a12cd84a1cc10 Mon Sep 17 00:00:00 2001 From: devswha Date: Wed, 2 Sep 2026 16:22:13 +0900 Subject: [PATCH] research(ko): record the step-1 miss-review design decision (GO, measure-only) External review (GPT-5.6 Sol Pro, two messages) of roadmap step 1, the KO GPT-family miss-review manifest: GO. Records the row-level data contract (ko-gpt-miss-review.v1, hash-only, discovery-only provenance), the deficit-based root-cause taxonomy with an exclusive decision tree, the blinded two-reviewer procedure over the 56 current misses, the acceptance criteria, and the preregistration line any later treatment must cross. The reviewer's opinion and the maintainer session's judgment are kept separate; cited lines were spot-checked against the tree. No collector, manifest, threshold, pattern, lexicon, prompt, or runtime change. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_018ePnwo4nBWCWvFvpy1YgCN --- docs/research/humanization-data-backlog.md | 2 +- ...gpt-miss-review-step1-decision-20260902.md | 94 +++++++++++++++++++ 2 files changed, 95 insertions(+), 1 deletion(-) create mode 100644 docs/research/ko-gpt-miss-review-step1-decision-20260902.md diff --git a/docs/research/humanization-data-backlog.md b/docs/research/humanization-data-backlog.md index 7ab0bfe2..55bee99d 100644 --- a/docs/research/humanization-data-backlog.md +++ b/docs/research/humanization-data-backlog.md @@ -173,7 +173,7 @@ Do not commit private raw text when a hash, manifest row, or aggregate score is Execution order frozen for the next performance-only cycle (2026-09-01): -1. **KO GPT-family miss-review manifest.** Classify the currently available misses (up to 100) by register, deterministic signal breakdown, and root cause. Do not change thresholds or production behavior in this step. +1. **KO GPT-family miss-review manifest.** Classify the currently available misses (up to 100) by register, deterministic signal breakdown, and root cause. Do not change thresholds or production behavior in this step. **Reviewed 2026-09-02 — GO (measure-only); design, data contract, taxonomy, procedure and acceptance criteria in [`ko-gpt-miss-review-step1-decision-20260902.md`](./ko-gpt-miss-review-step1-decision-20260902.md); implementation deferred to its own branches.** 2. **Edited-AI intake and corpus.** Freeze light/heavy edit policies and the manifest schema before generating samples. 3. **Rewrite human-evaluation panel.** Run at least 30 randomized before/after pairs with 5 raters, separating naturalness preference from meaning-loss labels. 4. **Deterministic structure-transform experiment.** Test bounded merge/split and seam-only LLM infill against the single-pass baseline; keep it research-only. diff --git a/docs/research/ko-gpt-miss-review-step1-decision-20260902.md b/docs/research/ko-gpt-miss-review-step1-decision-20260902.md new file mode 100644 index 00000000..d67fd76d --- /dev/null +++ b/docs/research/ko-gpt-miss-review-step1-decision-20260902.md @@ -0,0 +1,94 @@ +# KO GPT-family miss-review manifest — step-1 review decision (2026-09-02) + +Status: **GO (measure-only)** — external review complete; no collector, manifest, taxonomy, threshold, pattern, lexicon, or runtime change was made in this step. +Roadmap: step 1 of the frozen performance-only order in `humanization-data-backlog.md`. Steps 2–8 stay inactive. + +This record separates two voices on purpose: what the external reviewer said, and what the executing maintainer session concludes from it. Nothing below changes production behaviour. + +## Review setup + +- Reviewer: GPT-5.6 Sol (Pro), via the Sol review lane, two messages in one conversation (message 1: verdict, data contract, traps; message 2: taxonomy, procedure, acceptance criteria). +- Context packed: 22 tracked files from a sanitized detached worktree of `dev` (tracked files only; `.env.example` removed from the worktree so the lane's secret guard passed). Exclusions: none from the handoff's required set. The pack was ~49.5K tokens. + - decision/evidence: `docs/research/humanization-data-backlog.md`, `2026-rebaseline.md`, `docs/benchmarks/rebaseline-audit-ko-latest.md`, `rebaseline-ko-latest.md`, `ko-confirmatory-verdict-20260901.md`, `ko-performance-improvement-handoff-20260818.md`, `2026-rewrite-efficacy.md`, `2026-rewrite-efficacy-study3.md` + - deterministic boundaries: `src/features/index.js`, `korean-diagnosis.js`, `korean-invariants.js`, `korean-structure-fingerprint.js` + - manifest/intake contracts and tests: `scripts/rebaseline-intake.mjs`, `rebaseline-build-claim-manifest.mjs`, `rebaseline-generate-modern.mjs`, `tests/quality/rebaseline-manifest.example.jsonl`, `tests/unit/rebaseline-intake.test.js`, `rebaseline-build-claim-manifest.test.js`, `rebaseline-generate-modern.test.js`, `korean-diagnosis.test.js`, `korean-invariants.test.js`, `korean-structure-fingerprint.test.js` +- Raw response artifacts and the conversation URL are kept in the maintainer-private records repository (`sol-reviews/`), not here. + +## What the reviewer said (message 1) + +**Verdict: GO.** The KO GPT-family cell misses 56 of 100 samples (`2026-rebaseline.md:74`: 44.0% catch, CI 34.7–53.8%), so measuring and decomposing that failure set before any change is the correct highest-priority task. + +**Row-level data contract** (public JSONL; every field required unless marked optional): + +| field | type / allowed values | invariant and source | +|---|---|---| +| `schema` | literal `ko-gpt-miss-review.v1` | versioned like `koDiagnosis.v1` (`korean-diagnosis.js:4`) | +| `sample_id` | non-empty, unique | copied from the scored manifest; duplicates are errors (`rebaseline-build-claim-manifest.mjs:66-71`) | +| `language` / `class` / `model_family` | literals `ko` / `ai-like` / `gpt-family` | the cell is never re-labelled | +| `register` | `blog` \| `academic-summary` \| `product-doc` \| `chat-update` \| `technical-how-to` | the generation matrix (`rebaseline-generate-modern.mjs:21`) | +| `provider` / `model` | verbatim from the row (currently `codex-cli` / `gpt-5.5`) | `rebaseline-generate-modern.mjs:32-37` | +| `generated_at` / `prompt_id` / `decoding` / `postprocess` | ISO date / string / object / object | copied losslessly (`rebaseline-generate-modern.mjs:229-241`) | +| `expected_hot` / `predicted_hot` | literal `true` / literal `false` | the mechanical definition of a miss (`rebaseline-audit-ko-latest.md:35-38`) | +| `patina_score` / `score_review` | number in [0,100] / object | copied, never recomputed during review | +| `redistribution` / `text_hash` | literal `hash-only` / `sha256:<64 hex>` | no `text` field; intake deletes private text and rejects hash mismatches (`rebaseline-intake.mjs:95-109`) | +| `source_review` | `{status, rationale}` non-empty | provenance for non-public rows (`rebaseline-intake.mjs:110-118`) | +| `source_doc` | literal `docs/research/2026-rebaseline.md` | claim surface | +| `source_manifest` / `source_manifest_hash` | manifest path / SHA-256 of its bytes before selection | freezes the corpus | +| `analysis_provenance` | `{git_commit, options_hash, analyzed_at, normalized_text_hash, signals_hash}` | analyzer options and NFC input fixed (`index.js:75-78`) | +| `analysis_role` | literal `discovery-only` | these hashes are never reused for performance validation | +| `reviewer` / `reviewed_at` | pseudonymous string / ISO-8601 | no real names | +| `taxonomy_version` / `miss_reason` | version string / `^[a-z][a-z0-9-]{0,63}$` from that version's code list | per-row reviewer reason is required (`humanization-data-backlog.md:50-53`) | +| `reviewer_notes` | 1–1000 chars, paraphrase only | no quotation, no distinctive phrases, no PII | +| `meaning_checks` (optional) | `{mps_proxy, fidelity}` numbers or null | meaning checks, never naturalness labels | +| `signals` | source-free projection produced by the fixed analyzer (document, paragraphs, advisory, diagnosis blocks) | valid only if `signals.document.hot === predicted_hot === false`; diagnosis signal IDs limited to the fixed set in `korean-diagnosis.js:16-33`; no source text, spans, or matched tokens stored | + +**Three most damaging traps and their controls:** +1. *Discovery-corpus reuse / threshold overfitting* — finding causes on these misses and then claiming improvement on the same hashes is circular. Control: `analysis_role: discovery-only`, frozen corpus/config hashes, and any later change evaluated only on a confirmatory corpus that excludes these `text_hash`es (consistent with `humanization-data-backlog.md:50-53, :166`). +2. *Register and model-family confounds* — KO catch differs by family (Claude 68%, Gemini 62%, GPT 44%; `2026-rebaseline.md:72-74`) and register FN spans 26.9–69.2% (`rebaseline-ko-latest.md:124-130`). Control: `register` and `model_family` are immutable fields; aggregate only per register; never generalize from this cell to all of KO or to other families. +3. *Unblinded label and private-text leakage* — a reviewer who already knows a row is a genuine false negative and sees text plus signals will rationalize; copying phrases into notes leaks the private corpus. Control: hide outcome/model/signal columns at first observation; public artifacts carry hashes, bounded scalars, rule IDs, and paraphrase only; MPS/fidelity stay in `meaning_checks`. + +## What the reviewer said (message 2) + +**Root-cause taxonomy.** Precondition for every row: `expected_hot=true`, `predicted_hot=false`, `signals.document.hot=false` (the document verdict is the OR at `src/features/index.js:215`); rows that violate it are invalid, not classified. For each row, freeze the active analyzer options (`burstinessBands`, `mattrBands`, `koDiagnosticBands`, `koEndingMonotonyBands`, `lexiconDensityThreshold`, `lexiconMinHotMatches`; `index.js:56-70`) and compute a normalized *gate deficit* per signal family (burstiness, MATTR, lexicon, KO diagnostics, structure): `max(0,(T−x)/|T|)` for `x ≥ T` gates, `max(0,(x−T)/|T|)` for `x < T` gates, `+∞` when the value or pattern is absent; AND gates take the max, OR gates and multiple paragraphs take the min. A review-only constant `NEAR = 0.10` is fixed in advance and never fed back into thresholds. Assign the first matching code, in this order: + +| `miss_reason` | rule | +|---|---| +| `multi-threshold-near` | two or more families within NEAR; record `near_families` and all margins | +| `threshold-near-burstiness` | only burstiness within NEAR (standard CV gate incl. its minimum sentence count, plus the KO ending-monotony AND gate; `index.js:131-159`) | +| `threshold-near-mattr` | only MATTR within NEAR (`index.js:136-137,170-173`) | +| `threshold-near-lexicon` | only lexicon within NEAR (density and hot-match count vs. their thresholds; impossible with zero pattern hits; `index.js:161-165`) | +| `threshold-near-ko-diagnostics` | exactly one KO diagnostic rule within NEAR, from its scalar and `koDiagnosticBands` (`index.js:139-144,173-175`; IDs serialized as `rhythm:`, `korean-diagnosis.js:24-26`) | +| `threshold-near-structure` | only structure within NEAR: min deficit over candor count/2, thematic-break count/3, structural-classifier score/threshold (`index.js:85-92,166-169`) | +| `threshold-far` | no family within NEAR but `min(d) < 1`; record `closest_family`, do not read it as a threshold fix | +| `advisory-only-coverage-gap` | all deficits ≥ 1 or ∞ and only translationese / post-editese rule counts exist (advisory signals deliberately outside `hot`, `index.js:93-98`) | +| `no-modeled-signal` | all deficits ≥ 1 or ∞ and no advisory rule either: the current signal surface has nothing that explains the miss | + +`skipReason` is never a root cause (skips are advisory; signals are computed unconditionally, `index.js:113-115`). Diagnosis signal IDs are only the code-generated set (`korean-diagnosis.js:18-33`). + +**Sampling and reviewer procedure.** (1) Freeze the population: SHA-256 of `rebaseline-2026.scored.public.jsonl`, then select `language=ko`, `class=ai-like`, `model_family=gpt-family`, `expected_hot=true`, `predicted_hot=false`. The cell currently holds 56 misses out of 100 (`2026-rebaseline.md:74`), so all 56 are reviewed with no sampling. (2) Only if the population ever exceeds 100: strata `(model_family, provider, model, register)`, one seat per non-empty register, remainder by Hamilton largest-remainder on miss proportions, within-stratum order by `SHA256(corpus_hash + "\0" + sample_id)`. (3) Two independent reviewers, each with their own `SHA256(corpus_hash + reviewer_id)` permutation; they see only `blind_id`, the scalar signal projection, the active gate settings, and the computed family margins. `sample_id`, register, provider/model, prior scores/outcomes, MPS/fidelity, and raw text are hidden; the reviewer locks `miss_reason` by the decision tree only. Original ID order is not a review order (generation IDs cycle through registers, `rebaseline-generate-modern.mjs:96-101`). (4) Raw text is never used for the taxonomy verdict. After locking, an authorized reviewer may consult the gitignored private corpus only for source-integrity checks or extraction anomalies, re-verifying `text_hash` first and logging `raw_text_accessed`, purpose, and reviewer; no copying, quoting, or new sidecars. (5) Disagreements: keep both original labels and margins with `disagreement=true`; recompute margins with the same code; regenerate on extraction/config errors; otherwise a third reviewer adjudicates from the same blinded view; `final_reason` and rationale are recorded separately; any unresolved row blocks the whole artifact. + +**Acceptance criteria (step 1 is done only when all hold).** Every current miss appears exactly once (56 today; the deterministic cap applies only above 100); each row carries signal breakdown, register, model family, and reviewer reason. Every row is bound to the source-manifest SHA, a unique `sample_id`, a valid `text_hash`, and the analyzer commit/options hash, and regenerating from the same inputs yields byte-identical signal JSON, margins, and `miss_reason`. The precondition triple holds on every row with no missing or NaN metric; no source text, sentence fragments, matched spans, or distinctive n-grams anywhere in the committed manifest or notes. Two independent labels, the initial agreement rate, a category confusion matrix, all original and adjudicated labels are recorded with zero unresolved disagreements. The summary reports population and selection counts and `register × miss_reason` and `provider/model × miss_reason` counts, without generalizing to naturalness or to all KO model families. The only permitted diff is schema, offline extractor/validator/tests, hash-only manifest, and the measure-only report; thresholds, score weights, patterns, lexicon, prompts, runtime, and production defaults do not change (`humanization-data-backlog.md:174-176, 185`). + +**Line to any later behaviour change.** A treatment may not start before a separate preregistration is committed and bound to a fresh corpus disjoint from the discovery `text_hash`es (as `ko-confirmatory-verdict-20260901.md:53-56` already requires). The preregistration must fix: the exact hypothesis and target `miss_reason`; the old→candidate value or the pattern/lexicon/prompt/runtime diff with code/asset/prompt/model/decoding hashes; independent calibration/confirmatory splits, corpus SHA, register/model strata, sample sizes, baseline, and stopping rule; for threshold or score changes, per-family/register catch rate and human-FPR limits with CIs (register-level review before tightening, `2026-rebaseline.md:85-87`); for pattern/lexicon changes, candidate selection rules, hot/cold lift, cold document frequency, context gates, and rejection rules (`humanization-data-backlog.md:75-78,169`); for prompt/runtime changes, paired blind naturalness, register fit, and clarity as separate primary outcomes with locked invariants, MPS/fidelity floors, and latency/token/cost budgets (`ko-performance-improvement-handoff-20260818.md:333-369`); and estimator, confidence level, tie/abstention/error and AB/BA handling, exclusions, missing data, multiplicity, and success/stop rules. Deterministic AI-likeness stays a secondary diagnostic for rewrite treatments, never a candidate-selection or promotion gate (`…handoff-20260818.md:371-375`). + +## Maintainer session's judgment + +The executing session spot-checked the reviewer's citations that carry the design (`index.js:215`, `:56-70`, `korean-diagnosis.js:16-33`, `2026-rebaseline.md:74`, `rebaseline-generate-modern.mjs:21,32-37`, `rebaseline-intake.mjs:95-109`, `humanization-data-backlog.md:174-176`); all resolve to the quoted code or text. The reviewer's GO agrees with the frozen order, and nothing in the review asks for a threshold, pattern, lexicon, prompt, or runtime change. + +Accepted as the step-1 design, with three clarifications the session adds: + +1. The taxonomy is computable: every `miss_reason` follows from the fixed analyzer output plus the active option values, so the extractor must emit the per-family deficits itself and the reviewer only confirms the tree. That makes the two-reviewer step cheap and keeps raw text out of the loop, which is the point. +2. `NEAR = 0.10` and the deficit normalization are review constants. They must be written into the schema (`taxonomy_version`) so a later change to the constant is a new taxonomy version, not a silent relabel. +3. The population is the 56 current misses; the stratified cap is documented for completeness but is not exercised now. If the scored manifest is regenerated before the extractor runs, the population is re-frozen from the new SHA and this record is amended, not silently reused. + +Sol's opinion and this judgment are kept separate above so the reader can see where the design came from. + +## Deferred implementation tasks (not started in this step) + +Each is its own branch and PR, measure-only, under the contract above: + +1. `scripts/ko-miss-review-extract.mjs` — select the frozen population from the scored public manifest, run the fixed analyzer, emit the source-free `signals` projection, per-family deficits, margins, and `miss_reason` per the decision tree; write the `ko-gpt-miss-review.v1` JSONL with `analysis_provenance` and `analysis_role: discovery-only`. +2. `scripts/ko-miss-review-validate.mjs` + unit tests — schema, precondition triple, hash/provenance binding, byte-identical regeneration, no-raw-text checks (mirroring the intake and claim-manifest validators). +3. Blinded review kit — per-reviewer permutation and the signal-only view; capture of both labels, margins, disagreement flags, adjudication. +4. Measure-only report — population/selection counts, `register × miss_reason`, `provider/model × miss_reason`, agreement rate, confusion matrix; committed under `docs/benchmarks/` with the manifest hash. +5. Only after 1–4: any treatment proposal goes through a new preregistration bound to a fresh corpus, per the line above.