feat(observability): persist LLM token usage and add sin-code tokens CLI

AGENTS.md

@@ -271,10 +271,11 @@ SIN-Code/
 │   │       ├── loopbuilder/   ← v3.4.0: shared factory (DRY)
 │   │       ├── vane/          ← v3.8.0: HTTP bridge to ItzCrazyKns/Vane (internal/vane)
 │   │       ├── stack/         ← v3.8.0: unified install/doctor across 3 layers
-│   │       ├── hub/           ← v3.12.0: static tool catalog
-│   │       ├── ledger/        ← v3.13.0: semantic session ledger (SQLite)
-│   │       ├── summary/       ← v3.13.0: deterministic session summary builder
-│   │       ├── llm/           ← provider layer
+ │   │       ├── hub/           ← v3.12.0: static tool catalog
+ │   │       ├── ledger/        ← v3.13.0: semantic session ledger (SQLite)
+ │   │       ├── summary/       ← v3.13.0: deterministic session summary builder
+ │   │       ├── usage/         ← v3.17.0 (#168): LLM token-usage persistence + aggregation
+ │   │       ├── llm/           ← provider layer (+ Recorder interface, #168)
 │   │       ├── orchestrator/  ← DAG, critic, adversary, governor, ...
 │   │       ├── memory/        ← (existing) store/search/embed
 │   │       ├── lsp/, notifications/, todo/, plugins/, sandbox/, attachments/, webui/
@@ -317,6 +318,11 @@ Lessons DB: `~/.local/share/sin-code/lessons.db` (SQLite, modernc).
 Goal Queue DB: `~/.local/share/sin-code/goals.db` (SQLite, modernc).
 Ledger DB: `~/.local/share/sin-code/ledger.db` (SQLite, modernc), overridable
 via `SIN_CODE_LEDGER`.
+Token Usage DB: `~/.local/share/sin-code/tokens.db` (SQLite, modernc,
+issue #168), overridable via `SIN_CODE_TOKENS_DB`. Stores one row per LLM
+call with prompt/completion/total tokens, model, source, session, and a
+USD cost derived from `llm.pricing_per_1k.*` overrides or the built-in
+price table (`internal/usage.DefaultPricing`).
 
 Headless JSON contract (stable API — never break without major bump):
 
@@ -346,6 +352,7 @@ Headless JSON contract (stable API — never break without major bump):
 | v3.14.0 | ✅ SHIPPED | Unified config subsystem (#34): `sin-code config init/show/validate`, expanded TOML schema, user + project deep merge, atomic writes, secret masking, 39 subcommands |
 | v3.15.0 | ✅ SHIPPED | Go-native SCA Phase 1 (#41), race-flake hardening (#59) |
 | v3.16.0 | ✅ SHIPPED | Forge integration (#37): `sin forge` command, `sin status` detection, 16th MCP tool in `mcp_config` |
+| v3.17.0 | ✅ SHIPPED | Token-usage persistence (#168): `internal/usage/` SQLite store + `internal/llm.Recorder` interface + `tokens show/tail/aggregate` CLI + `tokens.db` at `$XDG_DATA_HOME/sin-code/`. `sin-code summary` now ends with a `Tokens: N (in X / out Y) · est. cost: $Z` line. No fake numbers (session is silent until the first call). Hardcoded built-in price table for NIM/Anthropic/OpenAI/Fireworks; configurable per-model via `llm.pricing_per_1k."org/model" = USD` in `~/.config/sin/sin-code.toml`. |
 
 Each release tag ⇒ goreleaser builds linux/darwin/windows × amd64/arm64,
 updates `homebrew-sin` formula, and ships to GitHub Releases.
@@ -396,7 +403,7 @@ Each skill **must** contain:
 Skills ported from external repos (e.g. `Infra-SIN-OpenCode-Stack`) must include
 `lifecycle: external` and `sources:` in their metadata.
 
-### CLI subcommands (verified `cmd/sin-code/main.go`, v3.5.0)
+### CLI subcommands (verified `cmd/sin-code/main.go`, v3.13.0+ #168)
 
 ```
 Core:      discover, execute, map, grasp, scout, harvest, orchestrate,
@@ -407,8 +414,8 @@ Frontend:  serve, tui, webui
 Lifecycle: memory, knowledge, todo, notifications, orchestrator_run,
            orchestrator_agents, orchestrator_plan, update
 Utility:   read, write, edit, lsp, plugin, index, security, sbom,
-           config, self-update, hub, ledger, summary
-``` (v3.13.0: 39 subcommands, up from 36 in v3.9.0)
+           config, self-update, hub, ledger, summary, tokens
+``` (v3.17.0: 40 subcommands, up from 39 in v3.13.0)
 
 ### Hook events (verified `internal/hooks/hooks.go`, v3.5.0)
 
@@ -441,6 +448,7 @@ The Go binary writes SQLite DBs to two distinct on-disk locations:
 | Lessons log      | `cmd/sin-code/tui/.sin-code/lessons.db`            | `internal/lessons/store.go`        | #62    |
 | Session store    | `cmd/sin-code/tui/.sin-code/sessions.db`           | `internal/session/store.go`        | #62    |
 | Code index       | `cmd/sin-code/internal/.sin-code/index.bin`        | `internal/index_store.go`          | c06cf18 |
+| Token usage      | `$XDG_DATA_HOME/sin-code/tokens.db` (or `~/.local/share/sin-code/tokens.db`) | `internal/usage/store.go`    | #168   |
 
 These are ignored by `.gitignore` (lines 64–65). Do not `git add` them
 under any circumstances; the proper fix is to migrate to

CHANGELOG.md

@@ -4,6 +4,42 @@ All notable changes to the SIN-Code unified binary will be documented in this fi
 
 ## [Unreleased] - 2026-06-16
 
+### Added — Token-Usage Observability (issue #168)
+- **`internal/usage/`** — SQLite-backed token-usage ledger (CGo-free,
+  `modernc.org/sqlite`, single-writer via internal mutex, race-safe). One
+  row per LLM call with `session_id`, `model`, `source` ("chat" | "verify"
+  | "judge" | "summary" | "plan" | "adhoc"), `input_tokens`,
+  `output_tokens`, `total_tokens`, `cost_usd` (derived at write time from
+  `llm.pricing_per_1k` overrides or the built-in price table), and
+  `created_at`. DB lives at `$XDG_DATA_HOME/sin-code/tokens.db` (override
+  via `SIN_CODE_TOKENS_DB`) — never in a gitignored subdir.
+- **`internal/llm.Recorder` interface** — pluggable persistence surface
+  the LLM client invokes on every successful `Chat` (and on every
+  `provider_adapter` completion that bypasses `Client.Chat`).
+  `NopRecorder` is the default; `loopbuilder` wires the SQLite-backed
+  recorder once `usage.Open` succeeds. Threading: SessionID flows via
+  `context.Context` (`llm.SessionIDContextKey{}`) — agentloop sets it;
+  the LLM client reads it.
+- **`sin-code tokens`** (new subcommand, 39 → 40, issue #168):
+  - `tokens show [--session ID] [--today] [--month] [--cost] [--share] [--json]`
+  - `tokens tail [--session ID] [-n 20]`
+  - `tokens aggregate [--by day|month|model|source|session] [--json]`
+- **One-liner in `sin-code summary`** — every summary now ends with
+  `Tokens: N (in X / out Y)` and `Estimated cost: $Z` when usage has been
+  recorded (caveman discipline: absent until the first call — never fake
+  a number).
+- **`llm.pricing_per_1k`** config key — flat map for per-model
+  USD-per-1k-tokens. Both quoted and unquoted model keys accepted
+  (`llm.pricing_per_1k."gpt-4o" = 0.0050` or
+  `llm.pricing_per_1k.gpt-4o = 0.0050`). Built-in price table covers
+  NIM (NVIDIA, llama-3.3-70b etc.), Anthropic (claude-sonnet-4,
+  claude-opus-4, claude-haiku-4), OpenAI (gpt-4o, gpt-4o-mini, o1),
+  Fireworks (`fireworks/llama-3.3-70b`, developer-opencode
+  `accounts/fireworks/models/minimax-m3`).
+- **Default permission rules** — `tokens__show`, `tokens__tail`,
+  `tokens__aggregate` all default to `allow` (read-only local DB; no
+  network, no mutation). Fits M4.
+
 ### Added — Loop Engineering (decoupled completion authority)
 - **Stop-gate harness** (`internal/stopgate`): an independent completion
   authority consulted after the verify-gate passes. Hybrid mode runs

cmd/sin-code/internal/agentloop/loop.go

@@ -12,6 +12,7 @@ import (
 	"github.com/OpenSIN-Code/SIN-Code/cmd/sin-code/internal/hooks"
 	"github.com/OpenSIN-Code/SIN-Code/cmd/sin-code/internal/ledger"
 	"github.com/OpenSIN-Code/SIN-Code/cmd/sin-code/internal/lessons"
+	"github.com/OpenSIN-Code/SIN-Code/cmd/sin-code/internal/llm"
 	"github.com/OpenSIN-Code/SIN-Code/cmd/sin-code/internal/permission"
 	"github.com/OpenSIN-Code/SIN-Code/cmd/sin-code/internal/session"
 	"github.com/OpenSIN-Code/SIN-Code/cmd/sin-code/internal/verify"
@@ -252,7 +253,10 @@ func (l *Loop) Run(ctx context.Context, sess *session.Session, prompt string) (*
 				reqMsgs = compressed
 			}
 		}
-		resp, err := l.Completion(ctx, reqMsgs, tools)
+		// Issue #168: thread SessionID through ctx so the LLM client / the
+		// adapter capture the right session ID on every chat completion.
+		turnCtx := llm.WithSessionID(ctx, sess.ID)
+		resp, err := l.Completion(turnCtx, reqMsgs, tools)
 		if err != nil {
 			return nil, fmt.Errorf("turn %d: %w", turn, err)
 		}

cmd/sin-code/internal/agentloop/loop_recorder_test.go

@@ -0,0 +1,72 @@
+// SPDX-License-Identifier: MIT
+// Purpose: tests for issue #168 — the agent loop threads the SessionID
+// through context so the LLM client recorder sees the right session.
+package agentloop
+
+import (
+	"context"
+	"testing"
+
+	"github.com/OpenSIN-Code/SIN-Code/cmd/sin-code/internal/llm"
+	"github.com/OpenSIN-Code/SIN-Code/cmd/sin-code/internal/session"
+	"github.com/OpenSIN-Code/SIN-Code/cmd/sin-code/internal/verify"
+)
+
+// TestRun_ThreadsSessionIDInContext: every completion call receives a
+// context whose SessionIDFromContext == the run's session ID. This is the
+// load-bearing contract for the LLM-usage recorder (#168).
+func TestRun_ThreadsSessionIDInContext(t *testing.T) {
+	s := setupSession(t)
+	gate := verify.NewGate("poc",
+		func(ctx context.Context, ws string) (bool, string, error) { return true, "ok", nil },
+		nil)
+
+	var got SessionIDsByCall
+	loop := &Loop{
+		Gate:      gate,
+		Workspace: "/tmp",
+		Completion: func(ctx context.Context, msgs []session.Message, tools []ToolSpec) (*Completion, error) {
+			got = append(got, llm.SessionIDFromContext(ctx))
+			return &Completion{Text: "done", Raw: session.Message{Role: "assistant", Content: "done"}}, nil
+		},
+	}
+	if _, err := loop.Run(context.Background(), s, "hi"); err != nil {
+		t.Fatal(err)
+	}
+	if len(got) == 0 {
+		t.Fatal("never called Completion")
+	}
+	for i, sid := range got {
+		if sid != s.ID {
+			t.Errorf("call %d session id: got %q want %q", i, sid, s.ID)
+		}
+	}
+}
+
+// TestWithSessionID_IsContextIdempotent: empty session ID returns the
+// parent unchanged (no context value added).
+func TestWithSessionID_IsContextIdempotent(t *testing.T) {
+	parent := context.Background()
+	if !isSameCtx(llm.WithSessionID(parent, ""), parent) {
+		t.Error("empty SessionID should not wrap parent ctx")
+	}
+	wrapped := llm.WithSessionID(parent, "sess-1")
+	if llm.SessionIDFromContext(wrapped) != "sess-1" {
+		t.Error("with-session-id round-trip failed")
+	}
+}
+
+type SessionIDsByCall []string
+
+func isSameCtx(a, b context.Context) bool {
+	if a == b {
+		return true
+	}
+	type keyer struct{}
+	aVal, _ := a.Value(keyer{}).(string)
+	bVal, _ := b.Value(keyer{}).(string)
+	_ = aVal
+	_ = bVal
+	// Different pointer, equal values? Compare both nil.
+	return a == b
+}

cmd/sin-code/internal/agentloop/provider_adapter.go

@@ -1,7 +1,8 @@
 // SPDX-License-Identifier: MIT
 // Purpose: bridges internal/llm.Client (STRUCT) to the agentloop.Completion
 // func signature via a func-closure (issue #44). Adds OpenAI-compatible
-// tool calling on top of the plain-chat client.
+// tool calling on top of the plain-chat client. Persists parsed token usage
+// via the client Recorder (issue #168).
 package agentloop
 
 import (
@@ -11,6 +12,7 @@ import (
 	"fmt"
 	"io"
 	"net/http"
+	"os"
 
 	"github.com/OpenSIN-Code/SIN-Code/cmd/sin-code/internal/llm"
 	"github.com/OpenSIN-Code/SIN-Code/cmd/sin-code/internal/session"
@@ -53,6 +55,14 @@ type wireResponse struct {
 		} `json:"message"`
 		FinishReason string `json:"finish_reason"`
 	} `json:"choices"`
+	// Issue #168: persist parsed Usage. Same shape as
+	// internal/llm.ChatResponse.Usage; kept local to avoid an import
+	// cycle (agentloop has no direct dep on llm.ChatResponse).
+	Usage struct {
+		PromptTokens     int `json:"prompt_tokens"`
+		CompletionTokens int `json:"completion_tokens"`
+		TotalTokens      int `json:"total_tokens"`
+	} `json:"usage"`
 }
 
 func NewProviderCompletion(c *llm.Client, model string, maxTokens int, temperature float64) func(ctx context.Context, history []session.Message, tools []ToolSpec) (*Completion, error) {
@@ -94,6 +104,16 @@ func NewProviderCompletion(c *llm.Client, model string, maxTokens int, temperatu
 		if err := json.NewDecoder(resp.Body).Decode(&out); err != nil {
 			return nil, fmt.Errorf("decode completion response: %w", err)
 		}
+		// Issue #168: persist parsed Usage via the client's Recorder so
+		// the agentloop completion path (which bypasses Client.Chat)
+		// captures the same Usage block. SessionID flows in via ctx.
+		if c.Recorder != nil && (out.Usage.PromptTokens != 0 || out.Usage.CompletionTokens != 0 || out.Usage.TotalTokens != 0) {
+			if recErr := c.Recorder.RecordUsage(ctx, llm.SessionIDFromContext(ctx),
+				model, llm.SourceChat,
+				out.Usage.PromptTokens, out.Usage.CompletionTokens, out.Usage.TotalTokens); recErr != nil {
+				fmt.Fprintf(os.Stderr, "warn: usage recorder: %v\n", recErr)
+			}
+		}
 		if len(out.Choices) == 0 {
 			return nil, fmt.Errorf("LLM returned no choices")
 		}

cmd/sin-code/internal/config.go

@@ -23,23 +23,24 @@ import (
 // namespaced keys (e.g. llm.base_url) for simple TOML-like parsing without
 // adding a parser dependency.
 type SinCodeConfig struct {
-	Theme            string   `toml:"theme"`
-	DefaultTimeout   int      `toml:"default_timeout"`
-	DefaultFormat    string   `toml:"default_format"`
-	MCPServerEnabled bool     `toml:"mcp_server_enabled"`
-	LLMBaseURL       string   `toml:"llm.base_url"`
-	LLMAPIKey        string   `toml:"llm.api_key"`
-	LLMModel         string   `toml:"llm.model"`
-	LLMMaxTokens     int      `toml:"llm.max_tokens"`
-	LLMTemperature   float64  `toml:"llm.temperature"`
-	AgentVerifyMode  string   `toml:"agent.verify_mode"`
-	AgentMaxTurns    int      `toml:"agent.max_turns"`
-	AgentHeadless    bool     `toml:"agent.headless"`
-	AgentYolo        bool     `toml:"agent.yolo"`
-	ToolsAllow       []string `toml:"permissions.tools_allow"`
-	ToolsDeny        []string `toml:"permissions.tools_deny"`
-	PathsMCPConfig   string   `toml:"paths.mcp_config"`
-	PathsSkillsDir   string   `toml:"paths.skills_dir"`
+	Theme            string             `toml:"theme"`
+	DefaultTimeout   int                `toml:"default_timeout"`
+	DefaultFormat    string             `toml:"default_format"`
+	MCPServerEnabled bool               `toml:"mcp_server_enabled"`
+	LLMBaseURL       string             `toml:"llm.base_url"`
+	LLMAPIKey        string             `toml:"llm.api_key"`
+	LLMModel         string             `toml:"llm.model"`
+	LLMMaxTokens     int                `toml:"llm.max_tokens"`
+	LLMTemperature   float64            `toml:"llm.temperature"`
+	LLMPricingPer1K  map[string]float64 `toml:"llm.pricing_per_1k"`
+	AgentVerifyMode  string             `toml:"agent.verify_mode"`
+	AgentMaxTurns    int                `toml:"agent.max_turns"`
+	AgentHeadless    bool               `toml:"agent.headless"`
+	AgentYolo        bool               `toml:"agent.yolo"`
+	ToolsAllow       []string           `toml:"permissions.tools_allow"`
+	ToolsDeny        []string           `toml:"permissions.tools_deny"`
+	PathsMCPConfig   string             `toml:"paths.mcp_config"`
+	PathsSkillsDir   string             `toml:"paths.skills_dir"`
 }
 
 func defaultConfig() SinCodeConfig {
@@ -53,6 +54,7 @@ func defaultConfig() SinCodeConfig {
 		LLMModel:         "",
 		LLMMaxTokens:     8192,
 		LLMTemperature:   0.0,
+		LLMPricingPer1K:  nil,
 		AgentVerifyMode:  "poc",
 		AgentMaxTurns:    80,
 		AgentHeadless:    false,
@@ -687,6 +689,18 @@ func applyMap(cfg *SinCodeConfig, m map[string]string) {
 			cfg.PathsMCPConfig = val
 		case "paths.skills_dir":
 			cfg.PathsSkillsDir = val
+		default:
+			// llm.pricing_per_1k.KEY = F  (issue #168).
+			if strings.HasPrefix(key, "llm.pricing_per_1k.") {
+				mk := strings.TrimPrefix(key, "llm.pricing_per_1k.")
+				mk = strings.Trim(mk, `"`) // accept both quoted and unquoted keys
+				if cfg.LLMPricingPer1K == nil {
+					cfg.LLMPricingPer1K = map[string]float64{}
+				}
+				if v, err := strconv.ParseFloat(val, 64); err == nil && mk != "" {
+					cfg.LLMPricingPer1K[mk] = v
+				}
+			}
 		}
 	}
 }

cmd/sin-code/internal/llm/provider.go

@@ -1,6 +1,7 @@
 // SPDX-License-Identifier: MIT
 // Purpose: generic OpenAI-compatible LLM client. Single-shot chat completion
 // request with bearer auth, JSON marshaling, and typed response decoding.
+// Persists parsed tool-usage via a pluggable Recorder (issue #168).
 package llm
 
 import (
@@ -57,6 +58,12 @@ type Client struct {
 	// in tests that hand-build the struct (HTTP roundtrips still work,
 	// just without breaker protection).
 	breaker *circuitbreaker.Breaker
+
+	// Recorder persists parsed ChatResponse.Usage on every successful
+	// call. Defaults to NopRecorder (drained) when wired via NewClient.
+	// Issue #168: previously the Usage block was parsed but dropped at
+	// provider.go:42-46; it now flows through a pluggable recorder.
+	Recorder Recorder
 }
 
 func NewClient(baseURL, apiKey string) *Client {
@@ -74,7 +81,8 @@ func NewClient(baseURL, apiKey string) *Client {
 			Timeout:   120 * time.Second,
 			Transport: circuitbreaker.RoundTripper(http.DefaultTransport, br),
 		},
-		breaker: br,
+		breaker:  br,
+		Recorder: NopRecorder{},
 	}
 }
 
@@ -126,6 +134,16 @@ func (c *Client) Chat(ctx context.Context, req ChatRequest) (*ChatResponse, erro
 	if err := json.NewDecoder(resp.Body).Decode(&out); err != nil {
 		return nil, fmt.Errorf("decode response: %w", err)
 	}
+	// Issue #168: persist parsed Usage. Best-effort — a recorder failure
+	// never blocks the caller. SessionID comes from the context (set by
+	// agentloop.Loop, the tui/chat runner, or the spec author).
+	if c.Recorder != nil && (out.Usage.PromptTokens != 0 || out.Usage.CompletionTokens != 0 || out.Usage.TotalTokens != 0) {
+		if recErr := c.Recorder.RecordUsage(ctx, SessionIDFromContext(ctx),
+			req.Model, SourceAdHoc,
+			out.Usage.PromptTokens, out.Usage.CompletionTokens, out.Usage.TotalTokens); recErr != nil {
+			fmt.Fprintf(os.Stderr, "warn: usage recorder: %v\n", recErr)
+		}
+	}
 	return &out, nil
 }