Skip to content

Add/pin manual model pricing (with free-tier windows) from the Telemetry dashboard tab #65

Description

@nujovich

Summary

Add a small form to the Telemetry dashboard tab (the plugin surface embedded in the
Hermes shell) that lets a user add or pin a model's pricing by hand, including a
free-tier toggle with an expiry date. Hand-added entries must survive the
OpenRouter / Google AI Studio auto-refresh (they already do — see below), so this is
mostly about exposing an existing capability through the UI plus one genuinely new
concept: a time-boxed free tier.

This is a future/backlog feature — filing it now so the scope and the non-obvious
constraints are captured while they're fresh.


Motivation

Today pricing can only be edited by hand-writing ~/.hermes/telemetry/pricing.yaml.
There is no in-product way to:

  1. Set a price for a model the auto-refresh doesn't know (self-hosted, preview, a
    provider without a pricing API).
  2. Declare that a model is free for a limited time (promo / free-tier window) and
    have that automatically stop applying once the window closes.

Both are common when tracking real spend across many models.


What already exists (do NOT rebuild)

  • Manual entries already survive auto-refresh. refresh_pricing() is a per-model
    merge, not a whole-file replace: any model not listed in _meta.auto_models is
    treated as manual and never overwritten — divergences are only reported, not applied.
    See pricing_refresh.py:259-282. So "don't let the
    refresh touch it" == "write the entry and keep it out of auto_models". No new
    preservation mechanism is needed.
  • A flat/free $0 marker exists: _subscription: true marks a model as $0 that the
    refresh never auto-populates (pricing.py:210-211). It has no
    expiry
    — that's the gap this feature fills.
  • Cache/reasoning prices are already optional-with-derivation. If cache_read /
    cache_write are absent they're derived from input × <multiplier> using the
    file-level defaults block (pricing.py:378-400,
    multipliers at pricing.py:141-142); reasoning defaults to
    output (pricing.py:403).
  • A UI→YAML write pattern already exists on the standalone dashboard: the budget
    modal (dashboard/index.html:295-341) POSTs to
    api_budget_update() which validates and atomically writes budget.yaml
    (serve.py:2393-2466). This is the template for the new
    pricing writer.

Proposed solution

Surface

Plugin telemetry tab: dashboard/dist/index.js (frontend) +
dashboard/plugin_api.py (backend API).

Backend — two new endpoints in plugin_api.py

  1. GET /models — data source for the selector. Returns the union of:

    • _DEFAULT_PRICING keys (pricing.py),
    • models currently in pricing.yaml,
    • models observed in the DB (llm_calls / runs).

    The frontend selector must also accept a free-typed model id not in the list
    (combobox) — pricing an unknown model is the primary use case.

  2. POST /pricing — validates the payload and atomically writes the entry into
    pricing.yaml (clone the api_budget_update() pattern). On success, call
    pricing.reload_custom_pricing() so the change is live without a restart.

    Contract note: plugin_api.py sets PRAGMA query_only=ON
    (plugin_api.py:58) and is otherwise read-only.
    That pragma guards the SQLite connection; pricing.yaml is a separate file, so a
    YAML writer does not violate it. But this is the first mutation on the plugin
    API — call it out explicitly and re-check
    tests/test_dashboard_plugin_isolation.py.

Data model (pricing.yaml)

  • Write the entry and do not add it to _meta.auto_models ⇒ auto-refresh leaves it
    alone (reuses pricing_refresh.py:259-282).
  • New per-entry metadata: _free_until: "YYYY-MM-DD", paired with the existing
    _subscription: true.

Form — two modes

Switch OFF (priced model):

  • input, output per 1M tokens — required.
  • cache_read, cache_write, reasoningoptional:
    • left empty ⇒ key omitted ⇒ runtime derives the default (input × multiplier, or
      output for reasoning). Existing behavior, no surprises.
    • filled ⇒ explicit override written to the entry.
    • UX: the empty field's placeholder shows the derived default live as input is typed
      (e.g. input=3.0, cache_read_multiplier=0.25 ⇒ placeholder 0.75), so the user
      sees what the default would be before deciding to override.

Switch ON (free tier):

  • Reveal a _free_until date picker; hide the price fields.
  • Writes { _subscription: true, _free_until: <date> }.

Free-tier expiry semantics (new logic in pricing.py lookup)

When an entry carries _free_until:

  • today ≤ _free_until ⇒ treat as $0 (subscription).
  • today > _free_until ⇒ ignore the free marker and fall through to the normal
    lookup chain (pricing.yaml price → _DEFAULT_PRICING:free rule → prefix table →
    provider-assumed). No post-free price is captured in the form.

Testability gotcha: the lookup needs "today" (datetime.date.today()). Make the
reference date injectable/patchable so before/on/after-expiry can be tested
deterministically.

Frontend

Render the form inside TelemetryPage (dashboard/dist/index.js:292-327),
not as a slot: the shell only mounts the 4 catalogue slots
(manifest.json, registrations at
dashboard/dist/index.js:551-555) and none fit;
unknown slot names are silent no-ops. Reuse the existing api() helper
(dashboard/dist/index.js:30) extended for POST. On save:
POST → refetch.


Risks / costs

  • The plugin frontend is a prebuilt React IIFE bundle with no JSX source in the repo
    (React.createElement, dashboard/dist/index.js). Adding a form means either
    hand-editing the compiled bundle or locating/restoring the build pipeline. This is the
    most uncomfortable cost and should be resolved first.
  • First write endpoint on an intentionally read-only plugin API — see contract note
    above; keep the DB connection read-only, write only YAML.
  • Validation matters: reject bad/empty model ids, malformed dates, negative prices, and a
    _free_until in the past (warn, still allow?).

Non-goals

  • No change to the auto-refresh mechanics themselves.
  • No editing of auto-refreshed entries from the UI (manual-only).
  • No post-free price captured in the free-tier mode (expiry falls back to normal lookup).
  • No DB schema change — pricing lives in YAML, so the migration rule does not apply.

Testing

  • pricing.py _free_until logic: before / on / after expiry, with an injected "today".
  • POST /pricing: writes a manual entry, does not add it to _meta.auto_models,
    atomic write, validation cases.
  • Round-trip regression: add via endpoint → run refresh_pricing() → entry survives
    untouched.
  • Isolation: honor HERMES_TELEMETRY_HOME / HERMES_HOME via paths.py.

Docs

Same-PR updates to ONBOARDING.md (pricing model, new _free_until metadata, the write
endpoint changing the plugin API's read-only character) and the README pricing section.

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions