Skip to content

research(ko): step-1 miss-review extractor, validator, blind kit, discovery manifest - #718

Merged
devswha merged 7 commits into
devfrom
research/ko-miss-review-extract
Sep 2, 2026
Merged

research(ko): step-1 miss-review extractor, validator, blind kit, discovery manifest#718
devswha merged 7 commits into
devfrom
research/ko-miss-review-extract

Conversation

@devswha

@devswha devswha commented Sep 2, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

Summary

Implements the deferred step-1 tooling from docs/research/ko-gpt-miss-review-step1-decision-20260902.md, commits the discovery manifest, the blinded two-reviewer labels and the measure-only report. No analyzer, threshold, pattern, lexicon, prompt or runtime change.

  • scripts/ko-miss-review-lib.mjs — schema ko-gpt-miss-review.v1, taxonomy ko-gpt-miss-taxonomy.v1 (NEAR = 0.10), per-family gate deficits from the production analyzer defaults, source-free signal projection, extraction, contract validation (precondition triple, hash/provenance/population binding, no Hangul or banned keys, byte-identical regeneration + 12-char leak window when the private corpus is present), blind-order permutation, label merge.
  • CLIs: ko-miss-review-extract.mjs (--on-drift fail|exclude), ko-miss-review-validate.mjs, ko-miss-review-kit.mjs (blind / adjudicate / merge), ko-miss-review-report.mjs (counts only). npm scripts benchmark:ko-miss-review:{extract,validate,report}.
  • docs/research/ko-gpt-miss-taxonomy-v1.md freezes the deficit encoding and resolves the two open readings (KO diagnostics as one AND gate; deficit 0 at equality).
  • artifacts/rebaseline-2025/ko-gpt-miss-review.v1.jsonl (48 hash-only rows, reviewed) + .exclusions.jsonl (8 rows).
  • docs/benchmarks/ko-gpt-miss-review-v1.{md,json} — population, sentence-count and closer-burstiness-gate tables, register × miss_reason, provider/model × miss_reason, closest family, family deficit summary, reviewer agreement + confusion matrix.
  • Decision record amended: population drift — 8 of the 56 frozen candidates now fire the KO ending-monotony gate added after the 2026-05-22 scoring (7 academic-summary, 1 blog); recorded as precondition-violated:document-hot exclusions, reviewed population 48. Re-scoring the public claim manifest (44% → 52% on this cell) is a claim-surface change left as an owner decision.

Results (counts only)

miss_reason n
threshold-far 44
threshold-near-burstiness 4

Closest family is burstiness on 43 of the 44 far rows. 32 rows have two sentences (generation design), so the standard burstiness gate is blocked by its 3-sentence minimum there and only the KO ending-monotony gate can reach burstiness; on the 16 three-sentence rows the standard cv gate is live (closer gate on 12, tied on 4). Population property, not a threshold recommendation.

Review: two blinded reviewers (reviewer-a = Claude agent session, reviewer-b = GPT-5.5 agent session; disclosed in the report) saw only blind ids, the signal projection, gate settings and margins. Initial agreement 48/48, no adjudication, all final labels equal the computed tree. The review verifies tree application from pre-computed family deficits, as clarification 1 of the decision record specifies.

Test plan

  • npm run lint (syntax, eslint, tsc, cspell)
  • npm test (unit + e2e; tests/unit/ko-miss-review.test.js uses synthetic KO fixtures in tmp dirs, no private corpus)
  • node scripts/ko-miss-review-validate.mjs --require-review with the private corpus present: population 56 = 48 + 8, regeneration 48/48 byte-identical, PASS
  • npm run check:no-private-assets
  • prose gate on all new/changed docs
  • local replica of the CI docs/benchmarks stale check: clean

Merge with a merge commit: every manifest row's analysis_provenance.git_commit points at eb3464d on this branch.

🤖 Generated with Claude Code

https://claude.ai/code/session_018ePnwo4nBWCWvFvpy1YgCN

devswha and others added 4 commits September 2, 2026 16:53
@devswha @claude
research(ko): step-1 miss-review extractor, validator, blind kit and …
eb3464d 
…report (measure-only)

Implements the deferred step-1 tooling from the 2026-09-02 decision record
without touching analyzer behaviour, thresholds, patterns, lexicon or prompts.

- scripts/ko-miss-review-lib.mjs: schema ko-gpt-miss-review.v1, taxonomy
  ko-gpt-miss-taxonomy.v1 (NEAR = 0.10), per-family gate deficits from the
  production analyzer defaults, the source-free signal projection, extraction
  from the frozen scored manifest + ignored private corpus, contract validation
  (precondition triple, hash/provenance binding, population binding, no
  Hangul/banned keys, regeneration + substring leak check when the private
  corpus is present), blind-order permutation and label merging.
- ko-miss-review-extract.mjs: --on-drift fail|exclude for rows the current
  analyzer now flags hot (recorded as exclusions, never classified).
- ko-miss-review-validate.mjs, ko-miss-review-kit.mjs (blind / adjudicate /
  merge), ko-miss-review-report.mjs (counts only).
- docs/research/ko-gpt-miss-taxonomy-v1.md freezes the deficit encoding and
  resolves the KO-diagnostics AND-gate and equality readings.
- tests/unit/ko-miss-review.test.js runs on synthetic KO fixtures in tmp dirs;
  no private corpus needed.
- package.json scripts benchmark:ko-miss-review:{extract,validate,report};
  artifacts/rebaseline-2025/.gitignore allowlists the two hash-only outputs.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018ePnwo4nBWCWvFvpy1YgCN
@devswha @claude
research(ko): step-1 discovery manifest — 48 reviewed rows, 8 precond…
fadfbcf 
…ition exclusions

Extracted with `node scripts/ko-miss-review-extract.mjs --on-drift exclude
--analyzed-at 2026-09-02T08:00:00Z` at commit eb3464d from the frozen scored
manifest (sha256:a8cf8565…) and the ignored private corpus.

- 56 KO GPT-family misses selected; 48 remain misses under the current
  analyzer and carry signals, margins and the computed miss_reason.
- 8 rows now fire the KO ending-monotony gate (added after the 2026-05-22
  scoring) and are recorded as precondition-violated exclusions, not labelled.
- Hash-only: no text, spans, matched tokens or Hangul in either file.
- Reviewer labels are not merged yet (review: null).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018ePnwo4nBWCWvFvpy1YgCN
@devswha @claude
research(ko): scope the miss-review leak check to free-text fields
815001a 
The substring leak check matched enumerated labels ("high", "clean") that
also occur as English words inside some private texts. It now scans only
reviewer notes, labels and adjudication rationale, and it slides a
12-character window so a quoted fragment embedded in a longer note is
caught; signal and margin blocks stay covered by the Hangul and banned-key
checks. Adds a positive leak test.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018ePnwo4nBWCWvFvpy1YgCN
@devswha @claude
research(ko): record population drift amendment for step 1
7d11ef5 
Eight of the 56 frozen candidates now fire the KO ending-monotony gate
(added after the 2026-05-22 scoring); they are recorded as exclusions and
the reviewed population is the remaining 48. Re-scoring the public claim
manifest is left as an owner decision.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018ePnwo4nBWCWvFvpy1YgCN
@vercel

vercel Bot commented Sep 2, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
patina Ready Ready Preview Sep 2, 2026 8:05am UTC

Request Review

@devswha @claude
research(ko): merge blinded step-1 labels and publish the measure-onl…
1c4b63f 
…y report

- Two independent blinded reviewers (pseudonymous `reviewer-a`, a Claude
  agent session, and `reviewer-b`, a GPT-5.5 agent session) labelled all 48
  rows from their own permuted sheets: signal projection, gate settings and
  margins only. Initial agreement 48/48; no adjudication needed; every final
  label equals the extractor's computed tree.
- Manifest rows now carry `review` (both labels, disagreement flag, final
  reason), `miss_reason`, `reviewer`, `reviewed_at`, `reviewer_notes`.
  Validation with --require-review: population 56 = 48 + 8, regeneration
  48/48 byte-identical, no Hangul, no leak window hit.
- docs/benchmarks/ko-gpt-miss-review-v1.{md,json}: population, register x
  miss_reason, provider/model x miss_reason, closest family, family deficit
  summary, agreement and confusion matrix. Counts only; reviewer kind is
  disclosed; no threshold recommendation.
- Backlog step 1 annotated; benchmarks README indexes the report.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018ePnwo4nBWCWvFvpy1YgCN
@devswha @claude
research(ko): report sentence-count and closer-burstiness-gate popula…
c6f8aba 
…tion tables

32 of 48 reviewed rows have two sentences, so the standard burstiness gate is
blocked by its 3-sentence minimum there; on the 16 three-sentence rows the
standard cv gate is live. Recorded as population properties, counts only.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018ePnwo4nBWCWvFvpy1YgCN
@devswha @claude
research(ko): note that tied OR sub-gates need no tie order in the ta…
ddc9c48 
…xonomy spec

Raised by a blinded reviewer: the standard and ending-monotony burstiness
gates can tie on the shared cv leaf. The family deficit is identical either
way; the report counts ties.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018ePnwo4nBWCWvFvpy1YgCN
@devswha
devswha merged commit fbcf9d6 into dev Sep 2, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@devswha