Structured failure logs in .dev-quality-logs/ for AI agent consumption (needs design)

[lipex360x]

Status: needs design. This issue captures the problem and the candidate options. No acceptance criteria are listed yet — the design pass will produce them.

Goal

When a dev-quality hook fails during pre-commit (or a direct check-all invocation), emit a structured machine-readable log per failure into a local, non-tracked directory. Downstream consumers — particularly AI coding agents driving the commit-until-green loop — should be able to read those logs and act on them surgically, instead of parsing stdout.

Concrete motivating case

A specialized AI subagent is being built to implement features in the consumer project docx_builder. The subagent follows the workflow through to commit, which means it must:

1. Stage files
2. Run git commit and observe pre-commit failures
3. Iterate fixes against each failed hook until everything passes

Today the only signal the subagent has is raw stdout from each hook — terminal output formatted for humans (ANSI colors, multi-line traces, indented sub-messages, inconsistent across hooks: ruff vs mypy vs bandit vs pylint C0103 all format differently). Parsing it reliably is brittle and the agent burns tokens re-reading the same noise across iterations.

Moving the responsibility of producing a structured failure record from each individual agent into the dev-quality tooling itself means:

• Every AI assistant (Claude Code, Cursor, Devin, Copilot Coding Agent) gets the benefit for free.
• The contract evolves once, in one repo, rather than N times in N agent prompts.
• A human can also use the log files post-hoc (bat .dev-quality-logs/latest/*.json) to inspect what failed in the last run.

Why this is a design issue, not a feature issue

Multiple forks need a decision before AC can be written.

1. Output format

Option	Pros	Cons
JSON	Universal parsing, jq-friendly, schema validation tooling	Less human-readable raw
YAML	Human-readable raw, consistent with .dev-quality.yaml	Less universal in tooling
JSONL (one violation per line)	Streamable, append-friendly	Less natural for nested data

2. File granularity

Option	Shape on disk
One file per failed hook per run	.dev-quality-logs/2026-05-23T14-30-22-ruff.json
One file per run, all hooks inside	.dev-quality-logs/2026-05-23T14-30-22.json (object with key per hook)
One file per failed hook + a latest/ symlink dir	Two-tier: dated archive + latest/ always pointing to the last run

3. Trigger scope

Option	Behaviour
Pre-commit only	Logs emitted only when invoked through the git hook path
Pre-commit + check-all direct runs	Both code paths emit logs
Always (every checker invocation)	Even individual check-abbrev path/ commands emit logs

The current check-all lives at the orchestration layer; individual hooks are standalone. The design must decide whether log emission is the orchestrator's responsibility or each checker's responsibility.

4. Opt-in vs always-on

Option	Risk
Always-on	Pollutes any project that hasn't gitignored the dir
Opt-in via .dev-quality.yaml (e.g. emit_logs: true)	Explicit, but slower adoption for AI workflows
Auto-on if .dev-quality-logs/ exists in the project	Self-bootstrapping; user creates the dir, tool starts logging

5. Retention policy

Option	Cost
Keep last N runs (e.g. 10)	Simple integer; predictable disk footprint
Keep last N days (e.g. 7)	Date-based; survives bursts of activity
Keep last failing run only	Cheapest; loses history
No retention; users clean manually	Zero logic; pile of files

6. Schema stability

If AI agents start consuming this format, it becomes a public contract. A schema version field (schema_version: "1") at the top of each file lets the format evolve without breaking consumers. Decision: include version from day one, or wait until v2 exists?

Candidate mechanisms (concrete shapes per fork)

Mechanism A — JSON, one file per hook per run, opt-in via config

project/
├── .dev-quality.yaml          # emit_logs: true
└── .dev-quality-logs/
    ├── 2026-05-23T14-30-22-ruff.json
    ├── 2026-05-23T14-30-22-mypy.json
    └── 2026-05-23T14-30-22-check_comments.json

{
  "schema_version": "1",
  "hook": "ruff",
  "timestamp": "2026-05-23T14:30:22Z",
  "exit_code": 1,
  "files_checked": ["docx_builder/builder.py"],
  "violations": [
    {
      "file": "docx_builder/builder.py",
      "line": 42,
      "column": 8,
      "rule": "E501",
      "message": "line too long (95 > 88 characters)",
      "suggestion": null
    }
  ],
  "raw_stdout": "docx_builder/builder.py:42:8: E501 line too long ..."
}

Mechanism B — Single JSON per run, all hooks combined

.dev-quality-logs/
└── 2026-05-23T14-30-22.json

{
  "schema_version": "1",
  "timestamp": "2026-05-23T14-30-22Z",
  "run_exit_code": 1,
  "hooks": {
    "ruff": { "exit_code": 1, "violations": [ ... ] },
    "mypy": { "exit_code": 0, "violations": [] }
  }
}

Mechanism C — Dated archive + latest/ symlink

.dev-quality-logs/
├── runs/
│   ├── 2026-05-23T14-30-22/
│   │   ├── ruff.json
│   │   └── mypy.json
│   └── 2026-05-23T14-25-10/
└── latest -> runs/2026-05-23T14-30-22/

AI agents always read .dev-quality-logs/latest/*.json and never need to compute the most recent timestamp.

Open questions to resolve in the design pass

1. Format: JSON, YAML, or JSONL?
2. Granularity: one file per hook, one file per run, or dated dir + latest/ symlink?
3. Opt-in: config key, presence of .dev-quality-logs/, or always-on?
4. Trigger: pre-commit only, all check-all runs, or every checker invocation?
5. Retention: count-based, date-based, last-only, or none?
6. Schema versioning: include from v1 or defer?
7. Should dev-quality provide a CLI to read the latest logs (dev-quality logs show)? Or is "agents read the files directly" enough?
8. How does the format handle hooks that don't produce per-violation output (e.g. shellcheck)? Capture stdout only?

Decision criteria

• Agent ergonomics — minimum-effort access for AI consumers. Reading latest/ruff.json beats globbing timestamps.
• Mental simplicity — a maintainer who's never seen the feature should understand the layout in under 30 seconds.
• Disk hygiene — the dir should not balloon into MB territory after a busy week.
• Backwards compatibility — adding this should not change current stdout behaviour. Logs are additive.
• Coverage cost — dev-quality enforces 100% coverage. The chosen mechanism should be small enough to fully test.

Design pass deliverable

• ADR file at docs/adr/0002-failure-logs.md (or next number) capturing the decision per fork plus rationale.
• Decision owner: lipex360x (maintainer).
• Trigger to begin the design pass: this issue gets a design/structured-failure-logs-N branch created.

Out of scope

• Streaming logs to a remote sink (Loki, Datadog, etc.). This is local-only.
• Aggregating logs across multiple repos. One project, one dir.
• Replacing stdout. Stdout stays exactly as-is for human readers.
• Building a parser/SDK for consumers. Consumers (agents, humans) parse the format directly.

Branch

This issue is needs-design — implementation does not start until the ADR Phase A artefact lands. The ADR PR uses:

git checkout main && git pull
git checkout -b design/structured-failure-logs-N

After the ADR merges, implementation gets a fresh feat/structured-failure-logs-N branch.

---

Final step — documentation and skill sync (mandatory for every issue)

Tick each item below by editing this issue body (gh issue edit <N>) as you land the corresponding commit. Do not batch ticks at the end.

Phase A — when ADR lands

• ADR committed at docs/adr/0002-failure-logs.md (or next number)
• AC produced from ADR added to this issue body
• Label changed from needs-design to enhancement

Phase B — when implementation lands

Phase B follows project TDD. The AC added to this issue at end of Phase A must be turned into failing tests first, then minimum implementation, then refactor. Concretely:

1. Red — for each AC item, add a failing test in the relevant test file under tests/. Commit: test: structured failure logs (failing).
2. Green — implement the minimum code in the affected checker / orchestrator. Commit: feat: structured failure logs.
3. Refactor — clean up duplication or clarity loss only. Tests do not change. Commit: refactor: structured failure logs.

Run quality gates between phases:

uv run pytest --cov=stacks --cov-report=term-missing
uvx --from . check-all .

Closure checks:

• CHANGELOG.md → entry added under new version (sets the version)
• skill/SKILL.md (Core) → updated if rule applies to all languages
• skill/python.md / skill/bash.md → updated if language-specific
• README.md → user-facing sections updated (new config key, new output dir, behavior)
• pyproject.toml + README badge → updated automatically via release.py
• Tag qualifies — new behaviour, new config key. Run uv run release.py --release after commit.
• git push after every commit