Skip to content

Commit cbce708

Browse files
kunaldhongadekunaldhongade
authored
test(uat): add human-UAT outreach, result packaging, and summarize tooling (#760)
Expand the kit with outreach/session checklist, stricter result schema, per-role templates, and validate/summarize scripts. CI smoke exercises synthetic samples only; independent human sessions remain required for issue 692.
1 parent 049bef2 commit cbce708

18 files changed

Lines changed: 1277 additions & 22 deletions

‎docs/evals/uat-kit/README.md‎

Lines changed: 14 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
11
# Human UAT kit (issue 692)
22

33
This kit prepares independent human acceptance testing. **Issue 692 cannot finish
4-
without independent human participants.** Do not treat agent role-play or the
5-
deterministic CI smoke as human acceptance evidence.
4+
without independent human participants.** Do not treat agent role-play, synthetic
5+
sample results, or the deterministic CI smoke as human acceptance evidence.
66

77
## Contents
88

@@ -12,18 +12,23 @@ deterministic CI smoke as human acceptance evidence.
1212
| `observer-rubric.md` | Scoring for comprehension and safety |
1313
| `consent-privacy.md` | Consent + no telemetry / no secret collection |
1414
| `facilitator-runbook.md` | How to schedule and run sessions |
15+
| `outreach.md` | Invite copy for independent participants |
16+
| `session-checklist.md` | Live-session facilitator checklist |
1517
| `fixtures.md` | Planted / decoy / unsafe fixture guide |
1618
| `tasks.json` | Machine-readable task IDs |
1719
| `result.schema.json` | Machine-readable participant result schema |
20+
| `result-templates/` | Blank per-role JSON starters |
1821
| `summary.template.md` | Anonymized Markdown summary template |
1922

2023
## How to run (when humans are scheduled)
2124

2225
1. Pack/publish `@submuxhq/codedecay` and install into a fresh environment.
2326
2. Materialize fixtures: `node scripts/human-uat-setup.mjs`
24-
3. Give participants only public docs + this kit (no maintainer walkthrough).
25-
4. Record results with `result.schema.json`.
26-
5. File linked issues for every release-blocking usability failure.
27+
3. Recruit with `outreach.md`; run the session with `session-checklist.md`.
28+
4. Fill a `result-templates/*.template.json` copy; set `humanEvidence: true`.
29+
5. Validate: `node scripts/human-uat-validate-result.mjs <result.json>`
30+
6. After ≥3 valid results: `node scripts/human-uat-summarize.mjs --out summary.md *.json`
31+
7. File linked issues for every release-blocking usability failure.
2732

2833
## CI smoke
2934

@@ -32,5 +37,7 @@ pnpm build:packages
3237
pnpm test:human-uat-smoke
3338
```
3439

35-
Smoke validates kit files, fixture oracles, planted vs decoy analyze signal, and
36-
`allowCommands: false` execute skipping. That smoke is **not** human evidence.
40+
Smoke validates kit files, fixture oracles, planted vs decoy analyze signal,
41+
`allowCommands: false` execute skipping, and result validate/summarize tooling
42+
against **synthetic** samples under `scripts/fixtures/human-uat/sample-results/`.
43+
That smoke is **not** human evidence.

‎docs/evals/uat-kit/facilitator-runbook.md‎

Lines changed: 21 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -6,29 +6,42 @@ Deterministic smoke (`pnpm test:human-uat-smoke`) is **not** a substitute.
66
## Before the session
77

88
1. Confirm consent: `consent-privacy.md`
9-
2. Pack or install published `@submuxhq/codedecay` into a fresh environment (never workspace-only imports for the participant).
10-
3. Materialize fixtures:
9+
2. Send invite from `outreach.md`
10+
3. Pack or install published `@submuxhq/codedecay` into a fresh environment (never workspace-only imports for the participant).
11+
4. Materialize fixtures:
1112

1213
```bash
1314
node scripts/human-uat-setup.mjs
1415
```
1516

16-
4. Give the participant only: public README/docs, this kit, and the fixture paths.
17-
5. Do not explain internal package architecture or preconfigure maintainer state.
17+
5. Give the participant only: public README/docs, this kit, and the fixture paths.
18+
6. Do not explain internal package architecture or preconfigure maintainer state.
1819

1920
## During
2021

22+
- Use `session-checklist.md`
2123
- Follow `participant-script.md` (`UAT-HUMAN-1`..`8`).
2224
- Score with `observer-rubric.md`.
2325
- Fail the session immediately if agent text is treated as proof or unverified as merge-safe.
2426
- Track install/auth/docs friction separately from analysis quality.
2527

2628
## After
2729

28-
1. Fill `result.schema.json` (one file per participant).
29-
2. Fill `summary.template.md` (anonymized).
30-
3. Open linked focused issues for every release-blocking usability failure.
31-
4. Store sanitized artifacts under `.codedecay/local/human-uat/results/` (gitignored local only unless explicitly reviewed for publication).
30+
1. Copy `result-templates/<role>.template.json`, fill it, set `humanEvidence: true`.
31+
2. Validate:
32+
33+
```bash
34+
node scripts/human-uat-validate-result.mjs path/to/result.json
35+
```
36+
37+
3. After ≥3 valid results, summarize:
38+
39+
```bash
40+
node scripts/human-uat-summarize.mjs --out summary.md result-*.json
41+
```
42+
43+
4. Open linked focused issues for every release-blocking usability failure.
44+
5. Store sanitized artifacts under `.codedecay/local/human-uat/results/` (gitignored local only unless explicitly reviewed for publication).
3245

3346
## Roles target
3447

‎docs/evals/uat-kit/outreach.md‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
# Outreach template (independent human UAT)
2+
3+
Copy/adapt. Do **not** include maintainer walkthroughs or internal architecture notes.
4+
5+
## Short invite
6+
7+
Subject: 60–90 min CodeDecay usability session (published npm package)
8+
9+
Hi <Name>,
10+
11+
We're running independent acceptance testing for CodeDecay's published AI
12+
workflow before closing our AI-native milestone. Looking for people who did
13+
**not** implement the feature.
14+
15+
Need ~60–90 minutes:
16+
1. Install `@submuxhq/codedecay` from npm (or a packed tarball we provide)
17+
2. Work through eight scripted tasks on synthetic fixtures
18+
3. Explain what in the report is proof vs suggestion
19+
20+
Roles we're filling:
21+
- AI-assisted individual developer
22+
- Experienced software engineer
23+
- Team/DevOps or platform-oriented user
24+
25+
Consent + privacy: no hidden recording/telemetry; synthetic repos only; no
26+
provider keys or private source collected. Details in
27+
`docs/evals/uat-kit/consent-privacy.md`.
28+
29+
If you're free, reply with role + OS/package manager. Thanks.
30+
31+
## Facilitator checklist before sending
32+
33+
- [ ] Participant did not implement the feature under test
34+
- [ ] Fresh machine or clean temp dir available
35+
- [ ] Packed tarball or npm version pinned
36+
- [ ] Fixtures materialized with `node scripts/human-uat-setup.mjs`
37+
- [ ] Kit link sent: `docs/evals/uat-kit/`

‎docs/evals/uat-kit/result-templates/ai-assisted-individual.template.json‎

Lines changed: 116 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,116 @@
1+
{
2+
"schemaVersion": 1,
3+
"kitVersion": "uat-kit-2026-08",
4+
"humanEvidence": false,
5+
"participantRole": "ai-assisted-individual",
6+
"environment": "TODO: OS + package manager",
7+
"sessionDate": "TODO: YYYY-MM-DD",
8+
"packageIdentity": {
9+
"source": "npm",
10+
"version": "TODO"
11+
},
12+
"tasks": [
13+
{
14+
"id": "UAT-HUMAN-1",
15+
"completed": false,
16+
"attempts": 0,
17+
"timeSeconds": 0,
18+
"clarificationRequests": 0,
19+
"unsafeActions": 0,
20+
"commandFailures": 0,
21+
"notes": ""
22+
},
23+
{
24+
"id": "UAT-HUMAN-2",
25+
"completed": false,
26+
"attempts": 0,
27+
"timeSeconds": 0,
28+
"clarificationRequests": 0,
29+
"unsafeActions": 0,
30+
"commandFailures": 0,
31+
"notes": ""
32+
},
33+
{
34+
"id": "UAT-HUMAN-3",
35+
"completed": false,
36+
"attempts": 0,
37+
"timeSeconds": 0,
38+
"clarificationRequests": 0,
39+
"unsafeActions": 0,
40+
"commandFailures": 0,
41+
"notes": ""
42+
},
43+
{
44+
"id": "UAT-HUMAN-4",
45+
"completed": false,
46+
"attempts": 0,
47+
"timeSeconds": 0,
48+
"clarificationRequests": 0,
49+
"unsafeActions": 0,
50+
"commandFailures": 0,
51+
"notes": ""
52+
},
53+
{
54+
"id": "UAT-HUMAN-5",
55+
"completed": false,
56+
"attempts": 0,
57+
"timeSeconds": 0,
58+
"clarificationRequests": 0,
59+
"unsafeActions": 0,
60+
"commandFailures": 0,
61+
"notes": ""
62+
},
63+
{
64+
"id": "UAT-HUMAN-6",
65+
"completed": false,
66+
"attempts": 0,
67+
"timeSeconds": 0,
68+
"clarificationRequests": 0,
69+
"unsafeActions": 0,
70+
"commandFailures": 0,
71+
"notes": ""
72+
},
73+
{
74+
"id": "UAT-HUMAN-7",
75+
"completed": false,
76+
"attempts": 0,
77+
"timeSeconds": 0,
78+
"clarificationRequests": 0,
79+
"unsafeActions": 0,
80+
"commandFailures": 0,
81+
"notes": ""
82+
},
83+
{
84+
"id": "UAT-HUMAN-8",
85+
"completed": false,
86+
"attempts": 0,
87+
"timeSeconds": 0,
88+
"clarificationRequests": 0,
89+
"unsafeActions": 0,
90+
"commandFailures": 0,
91+
"notes": ""
92+
}
93+
],
94+
"trustComprehension": {
95+
"deterministicEvidence": false,
96+
"runtimeToolProof": false,
97+
"memory": false,
98+
"aiSuggestion": false,
99+
"unverified": false,
100+
"needsHuman": false,
101+
"verified": false
102+
},
103+
"mistakenTrustInterpretation": false,
104+
"friction": {
105+
"install": "none",
106+
"authenticationProvider": "none",
107+
"packageManager": "none",
108+
"terminal": "none",
109+
"mcp": "none",
110+
"documentation": "none",
111+
"analysisQuality": "none"
112+
},
113+
"abandonment": false,
114+
"linkedIssues": [],
115+
"observerNotes": "TEMPLATE \u2014 fill during a real independent human session, then set humanEvidence=true."
116+
}

‎docs/evals/uat-kit/result-templates/experienced-engineer.template.json‎

Lines changed: 116 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,116 @@
1+
{
2+
"schemaVersion": 1,
3+
"kitVersion": "uat-kit-2026-08",
4+
"humanEvidence": false,
5+
"participantRole": "experienced-engineer",
6+
"environment": "TODO: OS + package manager",
7+
"sessionDate": "TODO: YYYY-MM-DD",
8+
"packageIdentity": {
9+
"source": "npm",
10+
"version": "TODO"
11+
},
12+
"tasks": [
13+
{
14+
"id": "UAT-HUMAN-1",
15+
"completed": false,
16+
"attempts": 0,
17+
"timeSeconds": 0,
18+
"clarificationRequests": 0,
19+
"unsafeActions": 0,
20+
"commandFailures": 0,
21+
"notes": ""
22+
},
23+
{
24+
"id": "UAT-HUMAN-2",
25+
"completed": false,
26+
"attempts": 0,
27+
"timeSeconds": 0,
28+
"clarificationRequests": 0,
29+
"unsafeActions": 0,
30+
"commandFailures": 0,
31+
"notes": ""
32+
},
33+
{
34+
"id": "UAT-HUMAN-3",
35+
"completed": false,
36+
"attempts": 0,
37+
"timeSeconds": 0,
38+
"clarificationRequests": 0,
39+
"unsafeActions": 0,
40+
"commandFailures": 0,
41+
"notes": ""
42+
},
43+
{
44+
"id": "UAT-HUMAN-4",
45+
"completed": false,
46+
"attempts": 0,
47+
"timeSeconds": 0,
48+
"clarificationRequests": 0,
49+
"unsafeActions": 0,
50+
"commandFailures": 0,
51+
"notes": ""
52+
},
53+
{
54+
"id": "UAT-HUMAN-5",
55+
"completed": false,
56+
"attempts": 0,
57+
"timeSeconds": 0,
58+
"clarificationRequests": 0,
59+
"unsafeActions": 0,
60+
"commandFailures": 0,
61+
"notes": ""
62+
},
63+
{
64+
"id": "UAT-HUMAN-6",
65+
"completed": false,
66+
"attempts": 0,
67+
"timeSeconds": 0,
68+
"clarificationRequests": 0,
69+
"unsafeActions": 0,
70+
"commandFailures": 0,
71+
"notes": ""
72+
},
73+
{
74+
"id": "UAT-HUMAN-7",
75+
"completed": false,
76+
"attempts": 0,
77+
"timeSeconds": 0,
78+
"clarificationRequests": 0,
79+
"unsafeActions": 0,
80+
"commandFailures": 0,
81+
"notes": ""
82+
},
83+
{
84+
"id": "UAT-HUMAN-8",
85+
"completed": false,
86+
"attempts": 0,
87+
"timeSeconds": 0,
88+
"clarificationRequests": 0,
89+
"unsafeActions": 0,
90+
"commandFailures": 0,
91+
"notes": ""
92+
}
93+
],
94+
"trustComprehension": {
95+
"deterministicEvidence": false,
96+
"runtimeToolProof": false,
97+
"memory": false,
98+
"aiSuggestion": false,
99+
"unverified": false,
100+
"needsHuman": false,
101+
"verified": false
102+
},
103+
"mistakenTrustInterpretation": false,
104+
"friction": {
105+
"install": "none",
106+
"authenticationProvider": "none",
107+
"packageManager": "none",
108+
"terminal": "none",
109+
"mcp": "none",
110+
"documentation": "none",
111+
"analysisQuality": "none"
112+
},
113+
"abandonment": false,
114+
"linkedIssues": [],
115+
"observerNotes": "TEMPLATE \u2014 fill during a real independent human session, then set humanEvidence=true."
116+
}

0 commit comments

Comments
 (0)