feat: skill usage analytics (invocation tracking, leaderboard, TUI+Web parity) - #56
Merged
Conversation
Consolidated design from multi-agent brainstorming session (3 architects, adversarial critique, second pass to resolve blocking points). Covers schema, cost attribution formula, phased file list, and reference signatures for the skill invocation analytics feature. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BfnXvrvdLuhGXmyEPt4DsF
Add invocation_by_path map on DataStore so a single session update (update_session, the file-watcher entry point) can rescan only the touched file and re-fold the invocation_stats aggregate, instead of requiring a full compute_invocations() rescan of every session. Write locks on the map and the aggregate are scoped to the swap itself, never held across an await, per ADR-001.
compute_invocations() (a slow, sequential full rescan) blindly replaced invocation_by_path with its own snapshot, so a concurrent refresh_invocations_for() write from the file watcher landing while the scan was mid-flight got silently discarded once the scan finished. Both run concurrently at startup (main.rs spawns compute_invocations() as a background task, then starts the watcher immediately after). Tag each invocation_by_path entry with a monotonic epoch. The full scan snapshots the epoch before it starts and merges its results instead of replacing the map, skipping any path whose entry is newer than that snapshot. Also drops the redundant sessions_analyzed reset in fold_invocations() flagged as a minor cosmetic note.
Extracts real Skill tool_use invocations from session transcripts (never textual mentions), persists them in two new additive SQLite tables (skill_scan_inventory, skill_events, CACHE_VERSION unchanged at 9), and exposes an exact leaderboard via `ccboard skills [--json] [--since] [--project] [--include-subagents] [<skill>]`. Token attribution mirrors session_index.rs's proportional base/remainder split exactly (zero changes to that file), verified by test_parsers_agree_on_token_totals. Subagent transcripts under a subagents/ directory are deduplicated by session path, never by the JSON sessionId they share with their parent. Cross-checked against the real ~/.claude/projects dataset with cc-skill-usage: flow-lean invocation count matches exactly and stably across repeated runs (9 invocations, 6 projects), both with and without --include-subagents.
Adds a dedicated Skills tab (crates/ccboard-tui/src/tabs/skills.rs) showing the invocation leaderboard from DataStore::skill_leaderboard(): invocation count, project count, total tokens, estimated cost, first/last seen, with a [retired] badge for skills that have invocation history but are no longer present in the on-disk skill catalogue. Also fixes the pre-existing gap where App::needs_refresh was set by poll_events() on every DataEvent but never consumed anywhere: the tick loop in lib.rs now re-fetches invocation_stats() and refreshes both the Agents invocation badges and the new Skills leaderboard whenever a refresh is flagged, instead of only once at startup. Two related fixes bundled in, both scoped by the design doc: classify_plugin() in plugin_usage.rs now tries an exact match on the full namespaced name before falling back to substring matching, preventing false positives across namespaces (e.g. local "brainstorming" vs "superpowers:brainstorming"). The Agents tab's Skills sub-tab now shows an explicit [unused] label gated on invocation_count == 0, matching the same source of truth as the new leaderboard tab.
Expose Phase 1's DataStore::skill_leaderboard() through a new Axum route (GET /api/skills/leaderboard) and a matching Leptos page at /skills, table parity only per the design doc scope. Each entry gets a retired flag computed fresh per-request against the on-disk ~/.claude/skills scan, matching the same source the TUI's SkillsTab uses for its own [retired] badge (confirmed via direct coordination with the TUI implementer) so both surfaces agree on which skills count as defined, including the shared blind spot around namespaced plugin skills.
…ssify_plugin no-op Fix round 1 response to code review: two Important findings, both confirmed. The [retired] badge on the new Skills leaderboard tab false-fired on any plugin-namespaced invocation (superpowers:*, vercel:*, ...) because the on-disk catalogue only ever held bare local .claude/skills/* names, never resolving installed plugin skills. Added scan_known_skill_names_blocking() in tabs/agents.rs, which walks both the plugins/marketplaces/**/skills/ and plugins/cache/**/skills/ trees collecting lowercased leaf skill names, and changed SkillsTab::is_retired() to match on the invocation's leaf name against that combined catalogue instead of the raw namespaced string. Also fixes the case-sensitivity bug (TDD vs tdd) as the same lowercase pass, and the staleness issue: the catalogue now rescans in the background (via a tokio mpsc channel, spawn_blocking, non-blocking poll each tick) whenever needs_refresh fires, instead of being scanned once at startup. classify_plugin()'s round-1 fix was a mathematically provable no-op (exact match implies substring match, so the added OR clause changed nothing). Restructured to check exact matches across both skills and commands before falling back to substring matching in either category, which actually changes behavior: an exact command match can no longer lose to an unrelated skill's substring collision. Replaced the non-discriminating test with one that fails against the pre-fix function and passes against the current one.
Three subagent task reports from .superpowers/sdd/ landed in earlier commits via a broad git add. They're internal planning artifacts, not shipped code; .gitignore now excludes the whole scratch workspace. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
on_disk_skill_names() only scanned ~/.claude/skills/, so any plugin-installed skill (superpowers:brainstorming, vercel:vercel-cli, ...) was falsely flagged [retired] on the web leaderboard even when actively used. Measured 23/94 rows wrong, including the highest-cost skill on the board (25 invocations, $8.02). Mirrors the TUI fix from c49b84b: walk plugins/marketplaces and plugins/cache recursively for skills/<name>/SKILL.md, build a lowercased leaf-name catalogue, and match invocations by their leaf (text after the last ':') instead of an exact full-name comparison. Corrects the doc comment above on_disk_skill_names() that claimed the TUI and web sides shared this blind spot, no longer true since c49b84b. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The plugin marketplace/cache walk touches hundreds of files on a populated ~/.claude; running it inline in the request handler blocks the tokio executor per this project's async conventions. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Pre-existing lint (clippy::useless_borrows_in_formatting), unrelated to this branch's feature work but flagged by CI's newer stable toolchain and blocking merge. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Same pre-existing lint as the previous commit, flagged by CI's clippy after fixing config.rs (clippy stops at the first error per crate). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Same pre-existing lint as the previous two commits, this time in the ccboard-types crate's mirrored copy of the same masked_api_key logic. Swept the whole workspace for remaining instances of this pattern, none found. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
FlorianBruniaux
added a commit
that referenced
this pull request
Aug 6, 2026
Skill usage analytics (PR #56): incremental invocation tracking, the ccboard skills CLI, and TUI/Web leaderboard parity. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Real Skill invocation tracking for ccboard, built off the design doc consolidated from a multi-agent brainstorming session (3 architects, adversarial critique, second pass on blocking points).
invocation_by_pathmap +refresh_invocations_for()), instead of only recomputing at startup. Includes a fix for a lost-update race between the incremental refresh and the background full scan (epoch-tagged entries, mutation-tested).session_index.rs's existing formula, two new additive SQLite tables (CACHE_VERSIONunchanged, purely additive schema), andccboard skills [--json] [--since] [--project] [--include-subagents] [<skill>].[retired]badge for skills with invocation history but no on-disk definition (correctly resolving plugin-installed skills, not just locally-installed ones), and a fix for a live-refresh gap that predated this feature (the TUI'sneeds_refreshflag was set but never consumed).Verification
session_index.rs: zero-line diff (measured project success criterion).CACHE_VERSIONstays at 9, schema additions are purely additive.ccboard skills --json --include-subagentscross-checked againstcc-skill-usage --no-cache --include-subagentsforflow-lean, exact match on invocation count and project count.cargo fmt --all && cargo clippy --all-targets && cargo test --allis clean except for one pre-existing, sandbox-specific test failure unrelated to this diff (live_monitor::tests::test_detect_live_sessions_no_panicshells out tops, a setuid binary this review environment's sandbox categorically refuses; confirmed via a dedicated sandbox-diagnostic pass, not present in real environments).Test plan
cargo fmt --all && cargo clippy --all-targets && cargo test --all--claude-hometest directory, live skill invocation appended to a watched session file, counter confirmed updating without restartcc-skill-usageforflow-leanmain🤖 Generated with Claude Code