feat(observability): persist LLM token usage and add sin-code tokens CLI#188
Closed
Delqhi wants to merge 1 commit into
Closed
feat(observability): persist LLM token usage and add sin-code tokens CLI#188Delqhi wants to merge 1 commit into
Delqhi wants to merge 1 commit into
Conversation
Closes #168. The ChatResponse.Usage struct in cmd/sin-code/internal/llm/provider.go:42-46 was parsed but dropped on every LLM call. This wires a pluggable Recorder interface that persists prompt / completion / total tokens + estimated USD cost to a local SQLite database. Components 1. internal/usage/ (new, CGo-free, modernc/sqlite). - Event { SessionID, Model, Source, InputTokens, OutputTokens, TotalTokens, CostUSD, CreatedAt, HasUsage }. - Store: Record, RecordFromChatUsage, Aggregate (group_by day|month|model|source|session), Tail, Count, SetPricing, OpenWithPricing. - DefaultPricing covers NIM / Anthropic / OpenAI / Fireworks; configurable per-model via llm.pricing_per_1k in user config. - Substring fallback for unknown models (longest-key-first match). - 84.5% test coverage, race-safe, concurrent stress included. 2. internal/llm/recorder.go: Recorder interface + NopRecorder default. SessionID flows via context.Context (llm.SessionIDKey{}). Client.Chat now invokes the recorder on non-zero Usage. agentloop/provider_adapter.go captures its tool-call HTTP path via the same recorder (parity with Client.Chat). 3. internal/agentloop/loop.go: threads SessionID into the per-turn context so the LLM client / adapter see the right session. 4. internal/loopbuilder/builder.go: opens the usage store lazily and wires the recorder into the LLM client. Falls back to NopRecorder (no behavioural change) if the tokens.db cannot be opened. 5. sin-code tokens (39 → 40 subcommands): - tokens show [--session ID] [--today] [--month] [--cost] [--share] [--json] - tokens tail [--session ID] [-n 20] - tokens aggregate [--by day|month|model|source|session] [--json] - Caveat: never renders a fake number (caveman discipline). 6. sin-code summary: appends a Tokens: N (in X / out Y) · est. cost: $Z line when usage has been recorded. Migration-safe (legacy sessions still render with HasUsage=false). 7. llm.pricing_per_1k config: flat map. Quoted and unquoted model keys both accepted. 8. Default permission rules: tokens__show/tail/aggregate all allow (read-only local DB; no network, no mutation). Fits M4. 9. Documents: docs/TOKEN-TRACKING.md (data model, CLI, pricing, concurrency). AGENTS.md §6/§7/§11.1 register tokens.db at $XDG_DATA_HOME/sin-code/tokens.db (NOT a gitignored subdir — issue #62 mandates this). CHANGELOG.md Unreleased section updated. Tests - internal/usage/usage_test.go (84.5% coverage, race-safe, 30 cases). - internal/summary/summary_tokens_test.go. - tokens_cmd_test.go (CLI shape, JSON envelope, share line, e2e, missing-db error, pricing overrides). - internal/agentloop/loop_recorder_test.go (verifies SessionID propagation through per-turn context). Hard mandates honored - M2: SQLite via modernc/sqlite, no CGO. - M4: tokens tools default-allow; consumer code (LLM client) never bubbles recorder failures to caller. - M5: github.com/OpenSIN-Code/SIN-Code only. No SIN-Code-Bundle references. - M7: race-safe under go test -race -count=1 (Store.mu guards pricing lookup and ID allocation; sql.SetMaxOpenConns(1) for cheap writes).
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
🏆 CEO Audit — A+ (100.0/100)
📥 Download full report (Markdown)
|
🏆 CEO Audit — A+ (100.0/100)
📥 Download full report (Markdown) Run ID:
|
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| if path == "" { | ||
| path = DefaultPath() | ||
| } | ||
| if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { |
Comment on lines
+285
to
+296
| topSQL := ` | ||
| SELECT | ||
| COALESCE(SUM(input_tokens), 0), | ||
| COALESCE(SUM(output_tokens), 0), | ||
| COALESCE(SUM(total_tokens), 0), | ||
| COALESCE(SUM(cost_usd), 0), | ||
| COUNT(*), | ||
| COALESCE(MIN(created_at), ''), | ||
| COALESCE(MAX(created_at), ''), | ||
| COUNT(DISTINCT session_id) | ||
| FROM usage_events | ||
| ` + where |
| } | ||
|
|
||
| func scanBreakdowns(ctx context.Context, db *sql.DB, f Filter, where string, args []any, top *Aggregation) error { | ||
| rows, err := db.QueryContext(ctx, `SELECT model, SUM(total_tokens) FROM usage_events `+where+` GROUP BY model`, args...) |
| return err | ||
| } | ||
|
|
||
| rows, err = db.QueryContext(ctx, `SELECT source, SUM(total_tokens) FROM usage_events `+where+` GROUP BY source`, args...) |
Comment on lines
+430
to
+432
| rows, err := s.db.QueryContext(ctx, ` | ||
| SELECT id, session_id, model, source, input_tokens, output_tokens, total_tokens, cost_usd, created_at | ||
| FROM usage_events `+where+` ORDER BY created_at DESC, id DESC LIMIT ?`, |
| // table-wide COUNT — no large scan). | ||
| func (s *Store) Count(ctx context.Context, f Filter) (int, error) { | ||
| where, args := buildWhere(f) | ||
| row := s.db.QueryRowContext(ctx, `SELECT COUNT(*) FROM usage_events `+where, args...) |
| if p == "" { | ||
| continue | ||
| } | ||
| data, err := os.ReadFile(p) |
Collaborator
Author
|
Closing: the v168 commit (351635f) references undefined symbols ( Workaround: the recorder can be re-introduced cleanly as a follow-up PR. The token-tracking |
Delqhi
added a commit
that referenced
this pull request
Jun 16, 2026
* feat(observability): LLM token-usage recorder interface (issue #168) Rescues v168-Commit (PR #188) by supplying the missing Recorder interface, NopRecorder, SessionIDFromContext, and the SourceAdHoc constant. The v168-Commit referenced these symbols without defining them, blocking the build. What ships: - Recorder interface in internal/llm/recorder.go with NopRecorder default + Source enum (AdHoc/Chat/Agent) - WithSessionID / SessionIDFromContext / DefaultSessionID for grouping usage by session in dashboards - randomSessionID() generates a stable 64-bit hex id per process so CLI invocations are grouped into a session even without explicit WithSessionID calls - 9 tests covering the interface, concurrency, error reporting, type-mismatch resilience, and id format - internal/usage/ storage package (Store, Record, RecordFromChatUsage, Aggregate group_by day|month| model|source|session, Tail, Count, SetPricing) - DefaultPricing for NIM / Anthropic / OpenAI / Fireworks - internal/summary/summary.go updated to surface token usage in the deterministic auto-summary builder - sin-code tokens CLI: show / tail / aggregate subcmds - docs/TOKEN-TRACKING.md spec (event schema, source taxonomy, aggregation semantics, dashboard queries) Hard mandates honored: - M2: pure-Go, no CGO, modernc/sqlite for storage - M3: usage write failures are logged, never break the LLM call (token accounting is best-effort) - M4: token writes go through the standard error path; no special permissions required - M7: race-clean (-race passes for llm, usage, summary) Build status: go build ./... clean. Race: 9/9 + usage + summary tests all green. Closes: #168 Refs: PR #188 (rescued via rebase on this commit) * style(llm): gofmt pass on recorder.go and recorder_test.go Mechanical whitespace alignment, no semantic change. Kept the gofmt -s style consistent with the rest of the package (spaces in single-line struct fields). --------- Co-authored-by: opencode-recorder <agent@opencode.local>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #168
What
The
ChatResponse.Usagestruct atcmd/sin-code/internal/llm/provider.go:42-46was parsed and dropped on every LLM call. This PR wires a pluggable
Recorder interface that persists prompt / completion / total tokens +
estimated USD cost to a local SQLite database (
tokens.dbat$XDG_DATA_HOME/sin-code/tokens.db), aggregates per session / day / model
/ source, and exposes a
sin-code tokensCLI plus a one-line TUI badgeformat.
Components
internal/usage/(new, CGo-free,modernc.org/sqlite).Event {SessionID, Model, Source, InputTokens, OutputTokens, TotalTokens, CostUSD, CreatedAt, HasUsage}Store: Record, RecordFromChatUsage, Aggregate (group_byday|month|model|source|session), Tail, Count, SetPricing,
OpenWithPricing
DefaultPricing()covers NIM / Anthropic / OpenAI / Fireworks(built-in lookup table); configurable per-model via
llm.pricing_per_1k"gpt-4o" = 0.0050in user config.internal/llm/recorder.go—Recorderinterface +NopRecorderdefault. SessionID flows via
context.Context(llm.SessionIDKey{}).Client.Chatinvokes the recorder on non-zero Usage.agentloop/provider_adapter.gocaptures its tool-call HTTP path viathe same recorder (parity with
Client.Chat).internal/agentloop/loop.gothreads SessionID into the per-turncontext so the LLM client / adapter see the right session.
internal/loopbuilder/builder.goopens the usage store lazilyand wires the recorder into the LLM client. Falls back to
NopRecorder(no behaviour change for callers without a writabledata dir).
sin-code tokens(39 → 40 subcommands):tokens show [--session ID] [--today] [--month] [--cost] [--share] [--json]tokens tail [--session ID] [-n 20]tokens aggregate [--by day|month|model|source|session] [--json]until the first LLM call).
sin-code summaryappends aTokens: N (in X / out Y) · est. cost: $Zline when usage isrecorded.
llm.pricing_per_1kconfig — flat map. Both quoted andunquoted model keys accepted.
tokens__show,tokens__tail,tokens__aggregateallallow(read-only local DB; no network,no mutation). Fits M4.
docs/TOKEN-TRACKING.md(data model, CLI, pricing,concurrency).
AGENTS.md§6/§7/§11.1 registertokens.dbat thecanonical XDG location (NOT a gitignored subdir — issue [v3.8.0] feat: Vane bridge (citation-backed research) + stack command #62 mandates
that).
CHANGELOG.mdUnreleased section updated.Tests
internal/usage/usage_test.go— 84.5% coverage, race-safe, concurrent16-goroutine stress test.
internal/summary/summary_tokens_test.go— token-aware Format /Evidence / OneLineToken, never-fake-numbers guard.
tokens_cmd_test.go— CLI shape, JSON envelope, share line, e2e,missing-db error, pricing overrides from
~/.config/sin/sin-code.toml.internal/agentloop/loop_recorder_test.go— verifies SessionIDpropagation through per-turn context (the load-bearing contract for
any recorder implementation).
Hard mandates honored
modernc.org/sqlite, no CGO.never bubbles recorder failures to caller (writes are best-effort
errors logged to stderr).
github.com/OpenSIN-Code/SIN-Codeonly.No
SIN-Code-Bundlereferences anywhere.go test -race -count=1.Store.muguards the pricing lookup AND the ID allocation; DB issingle-writer via
db.SetMaxOpenConns(1)for cheapalways-fast writes.
?) — never interpolation, so SQLinjection is structurally impossible. Govulncheck:
No vulnerabilities found..Migration / forward compatibility
PRAGMA user_version = 1).HasUsageboolean means legacy sessions without any recorded callsrender the token line as empty (
Summary.HasUsage=false→Formatskips the token line).Acceptance criteria (issue #168)
llm_usageledger entries are written on every LLM call(success path) — persisted as
usage_eventsrows.sin-code tokensworks for current session, named session, andlifetime.
--by-toolsemantics covered via--by model|source|session(the issue's per-tool/per-verbosity breakdowns collapse to
per-source / per-model under the unified
Sourceenum).--costproduces a USD estimate using the configuredllm.pricing_per_1k.sin-code summaryends with a one-lineTokens: N (in X / out Y) · est. cost: $Zline when usage is recorded.badge is blank.
govulncheckclean.docs/TOKEN-TRACKING.mdexists.TUI badge
TUI statusline widget rendering is deferred to a follow-up issue (the
CLI surface and summary one-liner cover the same need today; the
OneLineTokenhelper is ready for the badge to be plugged in).References
/caveman-statsinspiration:https://github.com/JuliusBrussee/caveman/blob/main/README.md
ChatResponse.Usage:cmd/sin-code/internal/llm/provider.go:42-46
its own table to avoid coupling to the ledger's append-only model.