Parity target: everything ddnet's PostgresLogger provides, so dual-write can end

[awksedgreep]

ddnet currently dual-writes DHCP/TFTP/NTP/TOD logs: structured records to Postgres (TimescaleDB hypertables) and formatted text to timeless-logs. Decision 2026-08-15: keep both during the product cycle (more logging = better), and treat this checklist as the bar timeless must clear for ddnet to retire PostgresLogger entirely. ddnet is the first demanding customer; this is its requirements list.

What Postgres provides today that timeless must match

• Structured field ingestion: ddnet ships operation, client_mac, client_ip, and Option-82 metadata (cable_modem_mac, circuit id) as first-class fields, and they are filterable server-side (client_mac:"aa:bb:..." works). The jsonline endpoint accepts the fields already; indexing/filtering them is the gap. (ddnet side: extend the VLogsBackend payload — small.)
• Field-level analytics: stats by (field) (timeless_logs#1), aggregates that survive busy windows (timeless_logs#2 work-row cap), so "top NAKed MACs in the storm hour" is one query.
• Retention + compression policy: per-stream age-based retention equivalent to the TimescaleDB policies ddnet runs (compress after 7d, drop after 90d), configurable without SQL migrations.
• Live tail: the enhanced-logs LiveView gets realtime updates via PubSub today; timeless needs a follow/tail mechanism the UI can subscribe to.
• UI story: log browsing/search pointed at timeless (timeless_logs_dashboard convergence) so the Postgres reader path has a successor.

What retirement buys ddnet when this lands

Single log store, no dual-write, no per-message Postgres inserts on the DHCP hot path (ddnet#4 becomes moot), no hypertable maintenance — and timeless gets proven against a 30k-device production workload before external customers see it.

Related: awksedgreep/ddnet#4 (interim: per-service min_log_level config exists if Postgres insert volume ever needs cutting before parity lands).

🤖 Generated with Claude Code

[awksedgreep]

Progress: two checklist items implemented (timeless-libsql 7d168d7, unreleased)

Field-level analytics — done. stats by (fields...) grouped aggregation landed in LogsQL with the full stats function vocabulary (count, count_uniq, sum, avg, quantile, histogram, …) supported grouped — the implementation partitions rows and runs the existing ungrouped kernel per partition, so grouped semantics are identical to ungrouped by construction. Output is deterministically ordered by group values; first N by (alias desc) gives bounded top-N. The motivating query is now exactly one line:

"DHCP NAK" _time:1h | stats by (client_mac) count() as hits | first 10 by (hits desc)

The busy-window half of this item was already fixed by the CLP pruning work on issue timeless-libsql#2: aggregate scans are charged by rows actually decoded, not window size, so storm-hour queries survive the work cap.

Retention + compression policy — done. TIMELESS_LOGS_RETENTION (<n> + required unit suffix s/m/h/d) applies to new and existing stores at startup — no SQL, no migration (new extension command retention: under the hood). Compression needs no policy at all: timeless compresses on the optimize cadence (~seconds after flush), so "compress after 7d" is strictly dominated. TimescaleDB-equivalent config is TIMELESS_LOGS_RETENTION=90d.

Structured field ingestion — server side done. TIMELESS_LOGS_INDEX_KEYS creates new stores with the allowlist and reindexes an existing store once at startup when the persisted allowlist differs, so adding client_mac to dhcp1's store is an env change + restart. Filters like client_mac:"aa:bb:…" already worked (scan path); with the key indexed they prune via postings. Remaining half is ddnet-side: extend the VLogsBackend payload to ship operation/client_mac/Option-82 fields (noted as small in the checklist).

Already done previously: live tail (/select/logsql/tail, shipped in 0.7.2 and running on dhcp1) and the UI story's first leg (timeless_logs_dashboard mounted behind auth in timeless_ui /admin/observability).

Checklist state

• Field-level analytics (stats by, work-cap fix)
• Retention + compression policy (env knobs, no SQL)
• Structured field ingestion — timeless side (indexing + startup reindex); ddnet payload extension outstanding
• Live tail
• UI story — log browsing pointed at timeless as the Postgres reader's successor (dashboard exists; convergence/UX call is ddnet's)

All of it ships with the next data-plane release alongside the CLP query speedups and /metrics.