Skip to content

feat: rare tool sessions — surface pi / deepseek / experimental harness activity #176

Description

@planetf1

Problem

pi and deepseek harness sessions are being collected and appear in the daily tool mix, but none of the analytics sections specifically surface them. Their model choices, costs, and what tasks they were doing are effectively invisible — you'd have to manually query spans or logs to find out what actually happened in those sessions.

What to build

Rare-tool session summary

A dedicated summary for tools that appear infrequently (pi, deepseek, and any future experimental harnesses), showing:

  • Session list: when, which model, total tokens, estimated cost, span name (task description)
  • Per-model token breakdown within rare-tool sessions
  • Comparison against your "main" tools to show relative cost/token efficiency

Storage

  • query_rare_tool_sessions() — sessions from tools not in {claude_code, opencode, codex, bob}; returns { tool, session_id, start_time, duration_ms, model, input_tokens, output_tokens, cost_usd, top_span_name }
  • "Rare" threshold: tools with fewer than N sessions in the window (configurable, default 10)

API

  • GET /api/genai/rare_tool_sessions

Web UI

  • New Analytics accordion: Rare Tool Sessions — table of sessions with tool badge, model, cost, and first span name as a task hint. Only shown when rare-tool data exists.

CLI

  • otelite usage --rare-tools — table of sessions, most recent first

Why it matters

  • Occasional pi and deepseek sessions are exploratory experiments; surfacing them makes the comparison deliberate rather than accidental.
  • When a rare tool is unusually expensive or slow compared to the baseline, this makes that visible without log-diving.

Acceptance criteria

  • GET /api/genai/rare_tool_sessions returns sessions for scope names outside the main-tool set
  • "Rare" threshold correctly excludes main tools
  • Web section only renders when rare-tool sessions exist (no empty accordion)
  • CLI renders a table
  • Unit test covers tool classification
  • All quality gates pass

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:cliotelite-cli cratearea:storageotelite-storage cratearea:webWeb dashboard UIpriority:p3Lower priority — nice to havetype:featureNew functionality

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions