Problem
Each OCR run produces rich structured data (result JSON with category/severity/path per finding), but our workflow does not extract or persist this data in a queryable form. To make data-driven decisions about OCR quality, concurrency, publication routing, and rule.json prompt tuning, we need structured telemetry across runs.
Currently the only data we have is ad-hoc manual analysis of individual run artifacts. There is no longitudinal view.
Proposed solution
Add a telemetry extraction step that runs after each OCR review and produces a structured summary:
Telemetry record
For each OCR run, emit a JSON object with:
{
"run_id": "...",
"pr_number": ...,
"sha": "...",
"ocr_version": "1.7.9",
"concurrency": 2,
"wall_clock_seconds": ...,
"files_previewed": ...,
"files_reviewed": ..., // derived from result
"total_findings": ...,
"findings_by_category": { "bug": N, "maintainability": N, ... },
"findings_by_severity": { "high": N, "medium": N, "low": N },
"findings_by_category_x_severity": { "bug/high": N, ... },
"inline_posted": ...,
"already_resolved": ..., // count of findings already addressed
"infrastructure_failure": false,
"policy_failure": false,
"file_read_failures": [...],
"token_usage": ... // if available
}
Storage
Two options (can be implemented independently):
Option A: GitHub Actions artifacts
- Upload the telemetry JSON as a named artifact on every run.
- Simple, no infrastructure, but requires manual download for analysis.
Option B: GitHub Actions step summary
- Render a compact markdown summary in the workflow run page (using
$GITHUB_STEP_SUMMARY).
- Immediate visibility in the Actions UI without downloading artifacts.
Dashboard
Once telemetry records exist, a simple script can aggregate them across runs to answer:
- Average finding count per run (and trend).
- Most common categories/severities (and trend).
- Average wall-clock time per run by concurrency setting.
- File-read failure rate.
- Inline comment volume over time.
This data directly informs #2672 (publication routing policy tuning) and #2676 (rule.json prompt iteration).
Acceptance criteria
Relationship to other issues
Problem
Each OCR run produces rich structured data (result JSON with category/severity/path per finding), but our workflow does not extract or persist this data in a queryable form. To make data-driven decisions about OCR quality, concurrency, publication routing, and rule.json prompt tuning, we need structured telemetry across runs.
Currently the only data we have is ad-hoc manual analysis of individual run artifacts. There is no longitudinal view.
Proposed solution
Add a telemetry extraction step that runs after each OCR review and produces a structured summary:
Telemetry record
For each OCR run, emit a JSON object with:
Storage
Two options (can be implemented independently):
Option A: GitHub Actions artifacts
Option B: GitHub Actions step summary
$GITHUB_STEP_SUMMARY).Dashboard
Once telemetry records exist, a simple script can aggregate them across runs to answer:
This data directly informs #2672 (publication routing policy tuning) and #2676 (rule.json prompt iteration).
Acceptance criteria
Relationship to other issues