Produce semantic snapshot seed for desktop-ui-cv grounding holdout

[mark-e-deyoung]

Purpose

Produce a small, reproducible real demo workload snapshot bundle from the existing WineBot harness for SemperSupra/desktop-ui-cv#29 (B2 grounding qualification holdout).

This is a capture/evidence task, not a model-evaluation task. Do not run KV-Ground or use candidate model output to choose/annotate targets.

Source authority

Pin/reconcile against WineBot commit:

7072813f8f888d95de1c8d4e94c6e40a6df1bf2f

Use the existing demo scripts and snap()/session artifact machinery unchanged where practical. If current main has moved, record the exact tested commit and reconcile semantic-equivalent snapshot labels before running; do not silently substitute a different workflow.

Observe / Discover

Before capture:

1. verify WineBot runtime/API/container prerequisites and API credential availability without printing secrets;
2. verify fresh_session, init_session, screenshot capture, session artifact path, and CV sidecar status;
3. record exact WineBot commit, Docker/runtime identity, host OS, screen resolution/scaling/theme, and whether CV enrichment is active;
4. do not mutate desktop-ui-cv grounding runtime or Model Foundry.

First capture slice

Run Customs Form and Notepad++ demos first.

Expected semantic snapshot labels from the pinned scripts:

demo/scripts/demo-customs-form.sh

• amazon_return_page
• acrobat_form_open
• acrobat_filled_form
• print_dialog
• package_ready

demo/scripts/demo-notepadpp.sh

• notepadpp_launched
• notepadpp_save_dialog
• notepadpp_file_verified

Do not add VLC/7-Zip merely to inflate sample count; their inspected scripts do not currently provide semantic snap() checkpoints. demo-cv-control.sh is an optional second increment only if the first eight snapshot states do not provide enough unique/query diversity.

Capture requirements

For every named snapshot retained:

• preserve the PNG exactly as captured;
• preserve source demo name + snapshot label + exact WineBot commit + session ID/reference;
• retain relevant snapshots.jsonl, expected-state/expected-element evidence, session metadata, and CV enrichment references when available;
• do not publish screenshots or session artifacts merely to make them easier to access;
• treat screenshot/session material as private by default until a separate authority decision says otherwise.

Deduplicate before annotation

The existing demo/output/real-val-frames is not sufficient: 30 filenames collapse to only two unique Git blobs. It remains provenance/smoke evidence only.

After capture, use the desktop-ui-cv PR #30 inventory helper (or semantic equivalent at its exact reviewed head) to compute relative path, size, PNG dimensions, SHA-256, and duplicate groups:

validation/grounding_holdout/inventory_pngs.py

Run with duplicate failure enabled. Do not interpret filename count as sample count.

Handoff artifacts

Provide a private/local handoff containing:

1. exact execution evidence (WineBot commit, commands, runtime/environment summary, PASS/FAIL per demo);
2. candidate PNG directory or private artifact location;
3. canonical candidate inventory JSON with SHA-256/dimensions/duplicate groups;
4. retained semantic snapshot labels and any excluded/failed labels with reasons;
5. provenance references to session metadata/annotation/CV files;
6. confirmation that candidate model output was not used to select or annotate the holdout.

Do not commit captured screenshots to a public repository as part of this issue.

Verify / exit

PASS when:

• both primary demos execute sufficiently to produce the expected semantic states, or failures are explicitly evidenced;
• candidate assets are inventoried and deduplicated before annotation;
• at least several materially distinct unique states remain after dedupe;
• provenance is sufficient for desktop-ui-cv#29 to create stable sample IDs and annotations;
• private handling is preserved;
• no model qualification has occurred yet.

If the primary demos cannot produce a sufficiently diverse unique seed, stop and report the evidence before adding another demo. The next candidate extension is demo-cv-control.sh, not arbitrary frame sampling.

Related consumer work: SemperSupra/desktop-ui-cv#29, draft PR SemperSupra/desktop-ui-cv#30.

[mark-e-deyoung]

Consumer-side provenance contract is now frozen

desktop-ui-cv draft PR #30 now includes the exact pre-annotation handoff contract at head 4c29051704737eaf48625920cbe3029b166cb14d.

After the WineBot capture/dedupe step, produce a private/local capture manifest conforming to:

• schemas/grounding-capture-manifest-v1.schema.json
• validator: validation/grounding_holdout/validate_capture_manifest.py
• operating sequence: validation/grounding_holdout/CAPTURE_HANDOFF.md

The manifest must bind:

• exact WineBot repository + full commit;
• harness reference and demos used;
• OS/screen dimensions/scaling/theme/runtime/CV-enrichment context;
• candidate inventory reference/counts;
• every retained unique semantic snapshot: demo, label, private asset ref, SHA-256, byte size, PNG dimensions, session reference;
• every excluded semantic checkpoint and reason;
• hard controls that semantic checkpoints were predefined, candidate model output was not used, retained images were not selected for training/tuning, dedupe occurred before annotation, and handling remained private.

The validator rejects duplicate retained SHA-256 identities, inconsistent inventory counts, mutable source refs, and weakened anti-leakage controls.

Required local validation after constructing the manifest:

python validation/grounding_holdout/validate_capture_manifest.py <capture-manifest.json>

(use the validator from desktop-ui-cv PR #30 exact head above or a reconciled later equivalent).

Only after that manifest passes and the retained set is frozen should desktop-ui-cv annotations be created. Candidate KV-Ground output must remain unseen until both capture selection and annotations are frozen.

No screenshots need to be published or committed to satisfy this contract.

[mark-e-deyoung]

Capture gate refinement before execution

A red-team review of the consumer-side PR #30 manifest found one pre-model falsifiability gap: the old manifest asserted that semantic checkpoints were predefined but did not bind the actual predefined checkpoint list, so a checkpoint could be silently omitted instead of retained or explicitly excluded.

A bounded corrective candidate now exists in SemperSupra/desktop-ui-cv:

• branch: fix/issue-29-capture-checkpoint-coverage
• based on PR fix: enforce lifecycle/control transition contracts #30 exact head 4c29051704737eaf48625920cbe3029b166cb14d
• current corrective head: d9b1560ce7e6857ebc8527cb16445c4c9505bc70

Before freezing a real #121 capture manifest, use the corrected semantics (or an exact reconciled equivalent):

1. declare the complete checkpoint set being attempted before capture selection/model output;
2. every declared (demo, snapshot_label) must appear exactly once as either a retained asset or an explicit exclusion with reason;
3. no late/invented labels;
4. no checkpoint may be both retained and excluded.

The planned first-slice labels in this issue do not change. This only makes their disposition mechanically complete and auditable.

No private hosted Actions were used for this corrective branch; keep actual screenshot capture private/local as already specified.