Add FERPA output discipline and automated name leak scanner

[chaz-clark]

Summary

Add output FERPA discipline (never display student names in conversation) and automated name leak scanner to prevent FERPA violations in AI agent conversations.

Background

Incident: 2026-07-28 FERPA violation in ds460-master grading conversation
RCA: ds460-master/grading/RCA_2026-07-28_FERPA_name_disclosure.md

Current FERPA discipline (canvas-toolbox/AGENTS.md) prevents reading .deid_master.csv for name lookup (input discipline), but doesn't prevent displaying names read from other files (output discipline).

Impact: Student names were displayed in conversation responses alongside grades, exposing FERPA Zone 2 data in LLM context and cloud conversation history.

Root Cause

Input/Output Asymmetry:

• Input discipline: "Don't read .deid_master.csv to look up names"
• Output discipline: MISSING — no rule preventing name display in conversation

AI agent read working grading files (_computed_grades.csv, _gradebook_canvas.csv) containing names, then included names in formatted responses without filtering.

Proposed Solutions

1. Output FERPA Principle (Documentation)

Add to canvas-toolbox AGENTS.md FERPA section:

Output discipline — NEVER display names in conversation:

Regardless of what files you read, NEVER include student names in conversational responses. Always use de-identified codes (user_id or deid_code) when referring to students.

Examples:

• CORRECT: "User 806485 received an A (95) as a compassion adjustment"
• CORRECT: "FR-B75C87 completed 2 KCs and 3 WCs, earning a B+ elevated to A"
• VIOLATION: "Esther Scott (806485) received an A"
• VIOLATION: "Student 806485 (Esther Scott, FR-B75C87) ..."

Why: Conversation logs are cloud surfaces. Names must never enter LLM context or conversation history.

2. Zone 2-Adjacent File Classification (Documentation)

Expand FERPA file list to clarify "never read" vs "read but don't display":

FERPA Zone 2 (NEVER READ):

• grading/.deid_master.csv
• grading/.known_names.txt
• grading/*/.keymap.json
• grading/**/.fetch_log.json
• grading/*/.review.csv
• grading/*/feedback/_grader*.csv
• grading/**/submissions_raw/**

FERPA Zone 2-Adjacent (CAN READ, NEVER DISPLAY NAMES):

• grading/*/_computed_grades.csv
• grading/*/_gradebook_canvas.csv
• grading/*/FINAL_REVIEW_COMMENTS_*.md
• grading/*/_actual_grades.csv

3. Automated Name Leak Scanner (Code Enhancement)

Add pre-output scanner to catch name leaks before conversation display.

Integration options:

• Claude Code pre-output hook (if supported)
• Post-hoc conversation audit script
• Pre-commit hook to scan agent session logs

4. FERPA-Clean Response Templates (Documentation)

Add templates for common grading queries to show proper FERPA-clean formatting.

Implementation Plan

Phase 1 (Immediate):

• Add Solution 1 (Output FERPA Principle) to canvas-toolbox AGENTS.md
• Add Solution 4 (response templates) to canvas-toolbox AGENTS.md
• Update ds460-master AGENTS.md to inherit from canvas-toolbox

Phase 2 (This Week):

• Add Solution 2 (Zone 2-Adjacent classification) to canvas-toolbox AGENTS.md
• Design Solution 3 (automated scanner) architecture

Phase 3 (Medium-Term):

• Implement Solution 3 (automated scanner)
• Test on ds460-master with name leak scenarios
• Deploy to all course repos using canvas-toolbox

Success Metrics

• Zero name leaks in conversation after implementation
• 100% FERPA-clean responses (user_id/deid_code only)
• Automated scanner catches violations before display
• Clear output discipline in documentation

Related

• Previous FERPA incident: 2026-07-01 (read .deid_master during accommodation)
• Previous FERPA incident: 2026-07-02 (displayed head -5 .deid_master)
• HG-5 incident: 2026-07-17 (KC3 grading protocol violation) — RCA in ds460-master

[chaz-clark]

Phases 1 and 2 shipped in #274 (v1.15.0). Keeping this open for Phase 3, which needs a redesign — the details below.

Shipped

• Zone 2-ADJACENT class (Solution 2) — _computed_grades.csv, _gradebook_canvas.csv, _actual_grades.csv, FINAL_REVIEW_COMMENTS_*.md. Read freely, never echo a name out of them.
• Unconditional output rule (Solution 1) — user_id/deid_code in every response, summary, table, and commit message, regardless of which file the name came from, with ✅/❌ pairs from the actual incident phrasing.
• Response templates (Solution 4) — folded into the ✅/❌ pairs rather than added as a separate block. A standalone template section would have restated the rule a third time in the same section; the examples carry it.
• The grading skill names the three working files it has you read and points back at the constitution.

Consumer repos inherit through the pointer block, so ds460-master picks this up on its next cb_update — the "update ds460-master AGENTS.md" checkbox resolves itself.

Phase 3 can't be built as specified

Add pre-output scanner to catch name leaks before conversation display.

There is no such seam. Claude Code hooks fire on tool calls (PreToolUse/PostToolUse), prompt submit, and stop — none of them can inspect or block assistant prose. The issue lists "Claude Code pre-output hook (if supported)" as an option; it isn't supported, and the other two listed options are both post-hoc, which is detection rather than the prevention the issue asks for.

Also worth recording: the automation gap is narrower than this issue assumed when filed.

• lib/tools/grader_name_leak_check.py already scans deid outputs against the roster with word-boundary matching, printing keys and counts only — never a name. That's the pre-cloud file gate.
• grade_guardian already blocks Zone-2 reads at both the Read tool and the shell (cat/head/tail/less/python open()), as of feat(grade_guardian): block shell reads of Zone-2 files; codify letters-are-read (#270) #270.

So what's actually missing is narrow: nothing nudges the model at the moment it reads a permitted file that happens to carry names.

Two options for a scoped-down Phase 3

1. PostToolUse hook on Zone-2-adjacent reads — fires when one of the four files is read and injects a reminder that it carries names and they must not be echoed. Prevention-shaped, cheap, fits the existing guardian architecture. Doesn't guarantee anything, but it puts the rule in context at the exact moment of risk rather than a thousand lines earlier in the constitution.
2. Post-hoc transcript audit — a script that scans a session transcript for roster names. Reliable detection, zero prevention; the leak has already reached the conversation by the time it fires. Useful as an audit trail, not as a control.

My read is that (1) is the only one worth building, and that it should be judged as defense-in-depth rather than as the guarantee "scanner" implies. Happy to build it on your word.

Suggested edit to the success metrics

• Automated scanner catches violations before display

This one isn't achievable and would leave the issue permanently unclosable. Suggest replacing with something like "the rule is present in context at the moment a name-bearing file is read."

[chaz-clark]

This enhancement would help significantly during course migration and setup work.

Current session context: Migrating Applied Calculus (MATH 119) from Spring 2026 (course 409936, 18 enrolled) to Fall 2026 (course 425166, 15 enrolled). Working with assignment dates, module structure, and grading scaffolding.

Why this helps:

• Course has grading assignments (Project tasks, AI logs, cohesive narratives) that will eventually contain student names
• Without output discipline, I could accidentally display names from _computed_grades.csv or _gradebook_canvas.csv when reporting grading setup status
• The "Zone 2-Adjacent" classification would clarify which grading files I can read vs. must never display

Specific value:

• Clear rule: "Use user_id/deid_code only, never names"
• Templates for FERPA-clean formatting during grading queries
• Pre-output scanner would catch violations before they reach conversation

+1 for implementing all four phases, especially Phase 1 (immediate documentation) and Phase 3 (automated scanner).