Skip to content

HANDOFF.md

Latest commit

 

History

History
391 lines (302 loc) · 15.9 KB

HANDOFF.md

File metadata and controls

391 lines (302 loc) · 15.9 KB
Raw

Suture — Handoff (fresh-agent bootstrap)

Public claims live in README.md (the paper) and docs/MODEL_CARD.md (Hub). Training order: TRAINING.md. This file is the operator log: jobs, gates, what not to do.

What this is: a fine-tune of Qwen3-VL-8B-Instruct (Apache-2.0; 27B is opt-in) that diffs an underwriting binder vs the issued policy → structured discrepancy report (policy-issuance QC). Specs: README.md, CONTRACTS.md (fixed benchmark contracts), docs/methodology.md (shared recipe across specula, suture, plumb, habeas).

Two PNGs go in (binder page, policy page). JSON comes out: PASS|FLAG + {type, severity, field, underwriting_value, policy_value, citation, correction} over the 13 CONTRACTS classes. The oracle is suture_forge.verify, not an LLM judge.

Repo map

forge/    suture_forge: seeded binder/policy generator + diff verifier-as-
          oracle + contamination monitor + CLI (pilot/split/leakprobe/
          benchmark/class-report)
model/    suture_model: dataset builder, SFT, GRPO, distill, ReST-EM,
          benchmark eval, schema.py = output contract + parser
cloud/    Modal L4 QLoRA (train/eval/rlvr) + AWS ReST-EM scripts
eval/     deterministic golden harness notes
docs/     DECISIONS.md, BENCHMARK.md, HANDOFF.md, MODEL_CARD.md,
          methodology.md, corpus/iso-aais-acord.md

JSONL under forge/data/ is gitignored. Regenerate with the CLI. Never commit .env, adapters, or *.jsonl.

Published model (2026-08-22 dedhi)

Adapter: /checkpoints/sft-dedhi/final on Modal volume suture-checkpoints. Hub: caiotheodoro/suture-8b. Do not overwrite sft-limithi/final or sft-ded/final. Base: Qwen/Qwen3-VL-8B-Instruct. 4-bit QLoRA, 384px, thinking off. Chain: sft-vl → sft-distill → sft-restem → sft-numeric → sft-limithi → sft-ded → sft-dedhi. Eval default in cloud/modal_eval.py is this adapter.

Seed-777 (split-line render) n=1000: recall 0.959 / HIGH 0.969 / prec 0.956 / parse 1.0 / FP/task 0.043. DEDUCTIBLE_HIGHER 0.957 / DEDUCTIBLE_LOWER 0.989. Val n=80: 1.000 / 1.000 / 0.970 (DED_HIGHER gold n=0 on val). JSONL /checkpoints/eval/sft-dedhi-bench777.jsonl.

CONTRACTS 0.95 rec/prec hold. HIGH 0.969, not 1.00. Limithi and sft-ded stay on the volume. Commit each gate; train/eval on Modal (spawn/detach), not a local GPU.

Previous published (2026-08-21 limithi)

Adapter: /checkpoints/sft-limithi/final on Modal volume suture-checkpoints. Was Hub caiotheodoro/suture-8b until dedhi. Seed-777: recall 0.839 / HIGH 0.893 / prec 0.870 / parse 1.0. DEDUCTIBLE_HIGHER 0.507. JSONL /checkpoints/eval/sft-limithi-bench777.jsonl.

Previous published (restem, loop closed earlier 2026-08-21)

Adapter: /checkpoints/sft-restem/final on Modal volume suture-checkpoints. Also on S3 s3://suture-restem-122862904842/restem/sft-restem/final/. Base: Qwen/Qwen3-VL-8B-Instruct. Training: 4-bit QLoRA, 384px thumbs, thinking off. Continue chain: sft-vl → sft-distill → sft-restem.

/checkpoints/sft-distill/final stays on the volume (777 recall 0.604). rlvr-g8 is not published (val recall dropped).

Eval default in cloud/modal_eval.py is the published adapter. Always modal run --detach (spawn). A non-detach modal run kills the GPU job when the local client exits.

Seed-777 n=1000 (contracted set, 739 FLAG, train overlap 0)

Model Recall HIGH Prec Parse FP/task
sft-dedhi (published, split-line) 0.959 0.969 0.956 1.0 0.043
sft-limithi (prior published) 0.839 0.893 0.870 1.0 0.118
sft-numeric (split-line) 0.788 0.854 0.805 1.0 0.176
restem split-line 0.634 0.795 0.704 1.0 0.231
sft-restem old pixels 0.622 0.756 0.691 1.0 0.247
sft-distill 0.604 0.733 0.665 1.0 0.269
GPT-5.6 Luna zero-shot 0.373 0.388 0.344 0.972 —
sft-vl 0.163 0.171 0.244 0.924 0.526
8B base 0.098 0.115 0.278 0.973 —

JSONL: /checkpoints/eval/sft-restem-bench777.jsonl (ap-UmZDzWw3MfKVr7FYpRgYZ3). Same scorer as distill (ap-WGkWnIl7FbemhBW0ZdnXf4).

Val n=80 (seed-7 holdout — optimistic, not the number)

Model Recall HIGH Prec Parse FP/task
sft-dedhi (published) 1.000 1.000 0.970 1.0 0.025
sft-limithi (split-line) 0.915 0.897 0.923 1.0 0.062
sft-numeric (split-line) 0.844 0.793 0.848 1.0 0.125
restem split-line 0.775 0.828 0.746 1.0 0.200
sft-restem old pixels 0.820 0.862 0.785 1.0 0.175
sft-distill 0.799 0.862 0.658 1.0 0.312
rlvr-g8 0.770 0.862 0.687 1.0 0.262

Citation EM ≈ 1.0 on SFT/ReST-EM rows is gold-string cloning, not a reading win. Do not report it as QC quality.

README targets remain 0.95 recall / 0.95 precision / 1.00 HIGH. This loop beat Luna on types and published a real 8B adapter. It did not hit the contract. Next loop exists to close that gap, starting at the classes that still look like Luna.

Seed-777 per-class (restem vs distill)

Diagnosis only. class-report --gold forge/data/benchmark.jsonl --pred <eval.jsonl>.

Class Distill rec Restem rec Restem prec
NAMED_INSURED 1.00 1.00 1.00
AI_ADDED 0.99 0.99 0.85
PERIOD_MISMATCH 0.80 0.85 0.97
PREMIUM_MISMATCH 0.81 0.82 0.86
COVERAGE_MISSING 0.79 0.77 0.78
FORM_ADDED 0.70 0.75 0.76
FORM_MISSING 0.55 0.61 0.66
COVERAGE_ADDED 0.49 0.49 0.40
AI_MISSING 0.44 0.46 0.72
DEDUCTIBLE_LOWER 0.21 0.22 0.68
LIMIT_HIGHER 0.15 0.20 0.24
LIMIT_LOWER 0.13 0.14 0.41
DEDUCTIBLE_HIGHER 0.10 0.07 0.16

ReST-EM bought form/period listing and fewer FPs. It did not teach limits/deductibles. DEDUCTIBLE_HIGHER got worse. Those three (plus COVERAGE_ADDED / AI_MISSING) are the next-loop surface.

Standing orders (do not)

  • Do not train on seed-777 / forge/data/eval-luna.jsonl / forge/data/benchmark.jsonl.
  • Do not mix RLVR traces into SFT.
  • Do not launch 27B. Default is 8B, L4, 4-bit.
  • Do not run more G=2 GRPO (collapsed). Do not run more G=8 GRPO on this L4 stack from restem (R3 dropped val 0.799 → 0.770).
  • Do not treat citation EM as the win.
  • Do not modal run GPU jobs without --detach / spawn().
  • Do not skip a git commit after a gate (mix leakprobe, val, 777, publish). Hub caiotheodoro/suture-8b tracks the published adapter only; overwrite it only after written 777 gates hold.
  • Do not kill plumb-r1 (g5.xlarge) — not this repo.
  • SageMaker ml.g6.2xlarge was CASE_OPENED; ReST-EM used EC2 instead.

Recall-lift loop (closed) — what actually happened

Recipe we ran, in order. Negative results stay in docs/DECISIONS.md.

R0 — error slice. Per-class precision/recall CLI (class-report). Luna 640 vs 384 aborted (OpenAI credits). 8B pass@8 skipped.

R1 — distill data. forge/data/pool-11.jsonl n=1200 (seed 11, signature-disjoint from 7 and 777). Mix train_mix.jsonl n=1520 = 320 seed-7 gold + 1200 seed-11 gold. reject_contaminated vs 777. Luna json_schema teacher never ran (credit_balance_exhausted); gold JSON was the teacher. Gold is a stronger type teacher than Luna 0.37; we lost Luna listing style.

R2 — distill SFT. Continue QLoRA from /checkpoints/sft-vl/final, 384px, 2 epochs, mix n=1520. Modal must spawn() not remote(). Val 0.799 gated. Seed-777 → 0.604. Adapter /checkpoints/sft-distill/final. Native-640 SFT cancelled at 38%; VRAM held, slower, not the leftover to reopen first.

R3 — G=8 GRPO. L4 held. TRL 1.10 use_dynamic_sampling. frac_reward_zero_std mean 0.26, never 1.0 (G=2 collapse did not recur). Val down 0.799 → 0.770. Do not publish. Do not 777-eval GRPO.

R4 / leftover — ReST-EM on AWS. Quota: EC2 G/VT On-Demand L-DB2E81BA = 8 vCPU. plumb-r1 held 4, so the job used g6.xlarge (1× L4 24GB) not g6.2xlarge. Pool = train.jsonl n=320 ∪ mix difficulty top 10% → 446 prompts, G=8, gold included as a candidate so SFT never trained worse-than-gold JSON. 1 epoch 384px continue from distill. Many kept labels were exact gold (reward delta 0); sampling still useful as gold-clone SFT on train+hard mix. Val 0.820 gated. Seed-777 0.622 → published.

AWS adapter: s3://suture-restem-122862904842/restem/sft-restem/final/. Account 122862904842, region us-east-1. Scripts: cloud/aws_restem.sh, cloud/aws_restem_start.sh. SFT_ONLY=1 skips sample and pulls restem.jsonl from S3.

attach_tokenizer_tokens(processor) after AutoProcessor.from_pretrained is required (TRL 0.17 died: Qwen3VLProcessor has no pad_token; pin trl>=1.10,<1.11, peft>=0.15,<0.18).

Environment

  • Modal: profile your-modal-profile. Volumes suture-checkpoints + suture-hf-cache. GPU L4. Always 4-bit.
  • AWS (ReST-EM leftover): us-east-1, bucket suture-restem-122862904842, role suture-restem-ec2, SG suture-restem-ssm. Instance must --instance-initiated-shutdown-behavior terminate (halt left a stopped box).
  • gcloud: you@example.com, project your-gcp-project, fallback only.
  • Forge .venv shebang has been broken; cd model && uv run pytest with PYTHONPATH=src:../forge/src.
  • uv + Python ≥3.11; macOS: make sync runs chflags -R nohidden .venv.

Eval (n=1000 is ~1.5–2h on L4)

modal run --detach cloud/modal_eval.py --data forge/data/benchmark.jsonl \
  --adapter /checkpoints/sft-restem/final --name sft-restem-bench777
# results: /checkpoints/eval/{name}.jsonl

Class report (local, no GPU):

cd forge && PYTHONPATH=src uv run python -m suture_forge.cli class-report \
  --gold data/benchmark.jsonl --pred /path/to/eval.jsonl

Next loop — HIGH = 1.00 (#12)

#13 published. Hub and eval default are sft-dedhi / caiotheodoro/suture-8b. DED rec gates are cleared.

Remaining contract hole is HIGH 0.969 vs 1.00. HIGH classes are COVERAGE_MISSING, FORM_MISSING, AI_MISSING, PERIOD_MISMATCH, NAMED_INSURED. Weakest on 777: AI_MISSING 0.912, FORM_MISSING 0.944. Do not start a mix until #12 writes the slice.

#3 ReST-EM is a later lever, not this pass.

#11 result (2026-08-22) — do not publish

gate bar sft-ded
val recall / HIGH ≥0.900 / ≥0.880 0.922 / 0.897 pass
777 recall / prec ≥0.86 / ≥0.90 0.886 / 0.941 pass
777 HIGH / parse ≥0.88 / 1.0 0.913 / 1.0 pass
DED_LOWER rec ≥0.70 0.897 pass
DED_HIGHER prec ≥0.70 0.700 pass
DED_HIGHER rec ≥0.70 0.507 fail

JSONL /checkpoints/eval/sft-ded-bench777.jsonl. Weights /checkpoints/sft-ded/final stay on the volume as an unpublished candidate.

Prior: deductible direction (issues #10–#11)

Numeric reading was published as sft-limithi (Hub then caiotheodoro/suture-8b). The remaining hole was sign, not unread digits.

limithi 777 numeric-slice (gold benchmark.jsonl, pred eval/sft-limithi-bench777.jsonl):

type tp fn fp fp_gold
LIMIT_LOWER 58 19 5 LIMIT_HIGHER 4, OTHER 1
LIMIT_HIGHER 58 23 26 LIMIT_LOWER 14, OTHER 9, PASS 3
DEDUCTIBLE_HIGHER 35 34 51 DEDUCTIBLE_LOWER 40, OTHER 11
DEDUCTIBLE_LOWER 43 44 2 DEDUCTIBLE_HIGHER 2

Diagnosis: 40/51 DEDUCTIBLE_HIGHER FPs are gold DEDUCTIBLE_LOWER (direction swap). Not PASS hallucination, not 5% pixels (DED is always ×2). Do not boost HIGHER alone — that would feed the FP.

Issues (strict order):

  1. #10 slice — this table.
  2. #11 contrastive DED SFT on Modal L4, continue limithi → /checkpoints/sft-ded/final.
  3. #12 HIGH only after DED gates.

Success (issue #11, not 0.95 in one shot)

Must all hold. Skip 777 if val fails.

  1. No 777 in train. leakprobe 0 vs val and benchmark.
  2. Val n=80 recall ≥ 0.900. Val HIGH ≥ 0.880.
  3. 777 DED_HIGHER and DED_LOWER recall each ≥ 0.70.
  4. 777 DED_HIGHER precision ≥ 0.70.
  5. 777 recall ≥ 0.86, precision ≥ 0.90, parse 1.0, HIGH not below 0.88.

Do not overwrite limithi or Hub until every row holds.

Numeric loop result (2026-08-21) — published as sft-limithi

sft-numeric missed LIMIT_HIGHER 0.35 (0.309). Seed-17 LIMIT_HIGHER mix n=2720 + one epoch → /checkpoints/sft-limithi/final. All gates held. Restem weights kept; eval default was limithi. Hub: https://huggingface.co/caiotheodoro/suture-8b

split restem new numeric limithi gate
val n=80 recall 0.775 0.844 0.915 ≥0.820 pass
777 recall 0.634 0.788 0.839 ≥0.70 pass
777 LIMIT_LOWER 0.078 0.740 0.753 ≥0.35 pass
777 LIMIT_HIGHER 0.111 0.309 0.716 ≥0.35 pass
777 DEDUCTIBLE_HIGHER 0.087 0.522 0.507 ≥0.35 pass
777 parse / prec 1.0 / 0.704 1.0 / 0.805 1.0 / 0.870 pass

Out of scope for the DED loop

  • Training on 777.
  • Boosting DEDUCTIBLE_HIGHER alone (feeds the 40 swap FPs).
  • Another undistinguished gold mix.
  • Native-640 SFT as the first lever (issue #7; DED is ×2).
  • Luna teacher traces (issue #4).
  • GRPO on this L4 stack (issue #6).
  • Qwen 2.4T (need a VL host; do not OCR-then-DeepSeek).

Hugging Face

Uploaded 2026-08-22: caiotheodoro/suture-8b is sft-dedhi (was limithi). Card is docs/MODEL_CARD.md.

Published:

  • adapter_model.safetensors + adapter_config.json + tokenizer / processor files from /checkpoints/sft-dedhi/final (≈200MB). Base Qwen/Qwen3-VL-8B-Instruct (Apache-2.0).
  • Honest 777 0.959 / 0.969 / 0.956 / 1.0, Luna 0.373, HIGH 0.969 not 1.00, citation EM is gold-clone, synthetic forge not carrier documents.

Do not publish:

  • Training JSONL (gitignored, regenerable from seeds).
  • A merged 16-bit 8B unless someone needs single-file vLLM without PEFT.
  • Any claim of general VL SOTA or “beats GPT-5 on insurance.” This is a closed synthetic QC task.

Re-upload the adapter only after a new checkpoint clears its written gates. Do not silently overwrite Hub with a val-fail run.

Bootstrap (fresh agent)

cd ~/Documents/personal/suture
make sync && make validate
cd forge && PYTHONPATH=src uv run python -m suture_forge.cli pilot --seed 7 --n 400 --out data/pilot.jsonl
cd forge && PYTHONPATH=src uv run python -m suture_forge.cli split --pilot data/pilot.jsonl --out-train data/train.jsonl --out-val data/val.jsonl
cd forge && PYTHONPATH=src uv run python -m suture_forge.cli leakprobe --train data/train.jsonl --eval-file data/val.jsonl
cd forge && PYTHONPATH=src uv run python -m suture_forge.cli leakprobe --train data/train.jsonl --eval-file data/benchmark.jsonl
cd forge && PYTHONPATH=src uv run python -m suture_forge.cli benchmark --seed 777 --n 1000 --out data/benchmark.jsonl --train data/train.jsonl

make validate then cd model && PYTHONPATH=src:../forge/src uv run pytest -q.

Prior status (2026-08-19, still true)

  • P0–P2: scaffold, forge, seed-7 400 → train 320 / val 80, seed-777 n=1000, leakprobe clean, Modal L4 always 4-bit.
  • P3 smoke: ap-ZgCz93ofVUzR7QxKLtC0BG SMOKE OK. 27B remains FULL_BASE_MODEL / --model.
  • P3 SFT: /checkpoints/sft-vl/final val recall 0.216 (ap-DL3Cgan9qPOUwFfVdAictx).
  • P4 GRPO-50 / 50b: miss. Keep SFT prior. G=2 closed.
  • P6: Luna 0.373 > SFT 0.163 > 8B base 0.098 on seed-777. DeepSeek unscored (text-only). That H2H is why the recall-lift loop ran.

Parallel repos

specula (food-label compliance — lead), plumb (AIA pay-app review), habeas (I-9 validation). Same scaffold/methodology; independently operable.