[bug] usage analytics của skills hình như chưa hoạt động

[mrgoonie]

Context

Usage Event Analytics đã có dashboard/resource tabs cho tool, skill, mcp_tool, runtime_tool, và backend có path ghi usage events cho skill activation.

Tuy nhiên khi sử dụng skills thực tế, phần usage analytics của skills trên dashboard hình như không tăng số đếm. Điều này làm dashboard không phản ánh đúng mức độ sử dụng skill, khó biết skill nào đang được invoke nhiều, skill nào lỗi, và skill nào cần cải thiện.

Observed behavior

• Kích hoạt/sử dụng skills nhưng dashboard Usage Analytics tab skill không thấy số đếm thay đổi.
• Có cảm giác skill usage events không được ghi, không được rollup, hoặc frontend/API đang không query đúng dữ liệu.
• Các tab khác như tool/runtime/MCP có thể vẫn hoạt động, nhưng skill analytics cần được kiểm tra riêng.

Expected behavior

Khi một skill được kích hoạt hoặc chạy qua các đường phổ biến, usage analytics phải ghi nhận được:

• skill name/slug;
• activation source, ví dụ use_skill, slash command, command invocation;
• call count;
• error count nếu có;
• channel/agent/user/team context nếu có;
• thời điểm/bucket để dashboard hiển thị theo range filter.

Dashboard Usage Analytics tab skill phải tăng số đếm sau khi skill được sử dụng và dữ liệu được refresh/rollup.

Places to inspect

Relevant areas found in current dev branch:

• backend skill usage event paths:
  • internal/agent/usage_events.go
    • recordToolUsageEvent(...) maps canonical tool use_skill to UsageEventTypeSkillActivation and UsageResourceTypeSkill
    • recordSkillSlashUsageEvent(...) records slash-command skill activation
  • internal/tools/use_skill.go defines use_skill as observability marker
• storage/rollup:
  • internal/store/pg/usage_event.go
  • internal/store/sqlitestore/usage_events.go
  • internal/tracing/snapshot_worker.go
• API/UI:
  • /v1/usage/events/summary
  • /v1/usage/events/breakdown
  • /v1/usage/events/timeseries
  • ui/web/src/pages/usage/hooks/use-usage-event-analytics.ts
  • ui/web/src/pages/usage/components/usage-event-analytics-panel.tsx

Recent related work:

• usage event analytics dashboard was added recently on dev.
• skill self-evolution metrics also has a separate skill_usage_metrics path, so confirm dashboard is not accidentally expecting that table instead of usage_events.

Reproduction steps

1. Start GoClaw on latest dev.
2. Open Usage Analytics dashboard.
3. Select the skill resource tab.
4. Trigger a skill in a normal chat/session, for example via use_skill + reading the skill, or a slash skill command if available.
5. Refresh dashboard after a reasonable delay / rollup interval.
6. Observe whether skill call count changes.

Possible causes to verify

• recordToolUsageEvent may not run for use_skill in some execution paths.
• tracing.TraceIDFromContext(ctx) may be nil, causing early return and dropping the event.
• Slash-command activation may use recordSkillSlashUsageEvent, but normal managed skill activation may not consistently pass through the same path.
• use_skill tool may be hidden/special-cased in a way that does not generate the expected tool result event.
• Usage events may be inserted, but hourly rollup/snapshot does not include the current bucket yet.
• UI may only query rollup tables, while recent events are not visible until the next hourly aggregation.
• resource_type=skill filtering may mismatch stored values.
• There may be confusion between usage_events and skill_usage_metrics.

Acceptance criteria

• Skill activation is recorded reliably for normal use_skill flows.
• Skill activation is recorded reliably for slash-command skill flows.
• Dashboard Usage Analytics tab skill shows updated counts for selected time range.
• API endpoints return non-zero rows for skill usage after a skill is used:
  • GET /v1/usage/events/summary?resource_type=skill&...
  • GET /v1/usage/events/breakdown?resource_type=skill&group_by=resource&...
  • GET /v1/usage/events/timeseries?resource_type=skill&...
• If rollup is intentionally delayed, dashboard should either:
  • include recent unrolled events, or
  • clearly communicate the delay so users do not think tracking is broken.
• Tests cover both use_skill and slash-command skill activation.
• Tests cover tenant isolation and channel/agent filters for skill events.
• No double-counting when a skill is activated once.

Notes

This is likely a tracking/rollup/API/UI integration bug, not a request to redesign the analytics UI. The first goal is to make the existing skill usage counters trustworthy.

[mrgoonie]

Root-cause investigation

I traced the current implementation and the skill event recording path itself appears to work. The issue is on the read/aggregation side: the dashboard/API reads only usage_event_rollups, while new skill events are inserted into usage_events and are not visible until the hourly rollup runs for a completed hour.

Evidence

1. Skill activation recording exists and has tests.

• internal/agent/usage_events.go

  • recordToolUsageEvent(...) maps canonical tool use_skill to:
    • event_type = skill_activation
    • resource_type = skill
    • source = use_skill
    • resource_name = args["name"]
  • recordSkillSlashUsageEvent(...) records slash-command skill activation as resource_type = skill.

• Focused tests pass on current dev:

go test ./internal/agent -run 'TestRecordToolUsageEvent_UseSkillCountsSkillName|TestRecordSkillSlashUsageEvent_RequiresTraceContext' -count=1 -v

Result:

=== RUN   TestRecordToolUsageEvent_UseSkillCountsSkillName
--- PASS: TestRecordToolUsageEvent_UseSkillCountsSkillName
=== RUN   TestRecordSkillSlashUsageEvent_RequiresTraceContext
--- PASS: TestRecordSkillSlashUsageEvent_RequiresTraceContext
PASS

So the basic use_skill -> usage_events mapping is not the obvious broken part.

2. The API used by the dashboard reads rollups only.

• internal/http/usage.go

  • /v1/usage/events/summary calls usageEvents.GetEventSummary(...)
  • /v1/usage/events/breakdown calls usageEvents.GetEventBreakdown(...)
  • /v1/usage/events/timeseries calls usageEvents.GetEventTimeSeries(...)

• internal/store/pg/usage_event.go

  • GetEventSummary(...) reads from usage_event_rollups
  • GetEventBreakdown(...) reads from usage_event_rollups
  • GetEventTimeSeries(...) reads from usage_event_rollups

Same pattern exists in SQLite store.

3. Rollups are only produced for completed hours.

• internal/tracing/snapshot_worker.go
  • catchUpUsageEvents(...) sets:

targetHour := now.Truncate(time.Hour).Add(-time.Hour)

That means events in the current incomplete hour are intentionally not rolled up.

4. Normal usage snapshots have current-hour live gap fill, but usage event analytics do not.

• internal/http/usage.go
  • normal /v1/usage/timeseries calls queryLiveHour(...) to include current incomplete hour from traces.
  • event endpoints /v1/usage/events/* do not have an equivalent live query against usage_events.

Root cause

Skill activations are written to usage_events, but dashboard/API reads only usage_event_rollups. Since rollups are generated only for the previous completed hour, current-hour skill usage is invisible in the dashboard. Users reasonably expect the Skill tab count to change after using a skill, but the UI has no live overlay for recent usage_events.

This is why it looks like skill usage analytics are not working even though the write path has tests and appears to record events.

Secondary risk

catchUpUsageEvents(...) uses GetLatestEventRollupBucket() and then advances from latest + 1h. If late events are inserted into an hour that was already rolled up, that hour may not be refreshed again by the normal catch-up loop. That can make some late-arriving skill/tool events permanently absent from rollups unless backfill/manual refresh is run.

Recommended fix direction

• Add live current-window aggregation for /v1/usage/events/* from raw usage_events, similar to queryLiveHour(...) for normal usage snapshots.
• Or change the event APIs to merge usage_event_rollups + raw usage_events for the incomplete/current bucket.
• Add tests proving that a newly inserted resource_type=skill event appears in summary/breakdown/timeseries without waiting for the next hourly rollup.
• Consider refreshing recent buckets, e.g. last 2-3 hours, instead of only latest + 1h, to handle late async inserts safely.

[Necmttn]

The live-overlay fix described above looks right. One extra guard I would add is event-level identity for skill activations.

Even if use_skill and slash commands both roll up as resource_type=skill, they should keep separate activation metadata:

• activation_source: use_skill | slash_command | command_invocation
• activation_id or tool_call_id
• trace_id / tenant / channel scope

That gives the API an exact dedupe key when it merges current-hour raw events with completed rollups. Without that, the dashboard can look fixed during the live window but still double-count or drop a skill at the rollup boundary.

---

Generated with ax.