Observational notes from two days of incident forensics against the production timeless-stack on ddnet dhcp1 (0.7.2, all three planes). Nothing here blocks; capturing per the file-everything policy.
Latency: unqualified pass
Every LogsQL query issued — dozens across two days — returned sub-second, including phrase scans over storm-hour data (6k+ matching rows), extract pipes, and count_uniq aggregations, all under live production write load (metrics + logs + traces ingesting simultaneously, oldest_queued_ms: 0 throughout). For a v0.7.x engine on real workload, response feel is already in the "instant" class that drives tool adoption. The known ceilings (work-row limit: timeless_logs#2, term intersection: timeless-libsql#2) are policy/planner issues, not raw-speed issues.
Storage efficiency at the same time: ~14.7 B/point (metrics, excluding WAL), ~29 B/log entry (~10x on repetitive DHCP logs), ~40 B/span vs ~232 B/span ingested.
Version stamping anomaly in the publish pipeline
On dhcp1's root podman storage: ghcr.io/awksedgreep/timeless-stack:latest (image ce7c1661, built ~13h before observation) self-reports 0.7.2 in /health — while 0.7.6-local (cf106bbb) was built an hour earlier. A :latest image built after 0.7.6 carrying 0.7.2 metadata suggests the publish pipeline stamps version from a stale source (old tag/env) rather than the built tree. Consequence: pull-and-restart on dhcp1 (2026-08-13) looked successful but still served 0.7.2. A timeless_build_info metric (timeless_stack#1) would make this visible on a dashboard; fixing the stamping makes it not happen.
Data-plane auth default caused silent ingest failure in dev
The dev instance's auth-on-by-default produced continuous HTTP 401 data_plane_authorization / missing_credentials from ddnet's log shipper — logs silently dropped rather than queued (client-side half tracked as ddnet#1). Known and being handled separately; recorded here because "default config silently loses data in the simplest local setup" is the kind of first-touch experience that costs early adopters.
Error taxonomy: positive note
{"error":"unsupported_capability","reason":"..."} with specific messages ("logical operator "or" is not implemented yet") is genuinely good DX — honest about limits, instantly actionable. The one inconsistency found (uppercase OR bypassing that honesty) is timeless_logs#3.
🤖 Generated with Claude Code
Observational notes from two days of incident forensics against the production timeless-stack on ddnet dhcp1 (0.7.2, all three planes). Nothing here blocks; capturing per the file-everything policy.
Latency: unqualified pass
Every LogsQL query issued — dozens across two days — returned sub-second, including phrase scans over storm-hour data (6k+ matching rows), extract pipes, and count_uniq aggregations, all under live production write load (metrics + logs + traces ingesting simultaneously,
oldest_queued_ms: 0throughout). For a v0.7.x engine on real workload, response feel is already in the "instant" class that drives tool adoption. The known ceilings (work-row limit: timeless_logs#2, term intersection: timeless-libsql#2) are policy/planner issues, not raw-speed issues.Storage efficiency at the same time: ~14.7 B/point (metrics, excluding WAL), ~29 B/log entry (~10x on repetitive DHCP logs), ~40 B/span vs ~232 B/span ingested.
Version stamping anomaly in the publish pipeline
On dhcp1's root podman storage:
ghcr.io/awksedgreep/timeless-stack:latest(image ce7c1661, built ~13h before observation) self-reports0.7.2in /health — while0.7.6-local(cf106bbb) was built an hour earlier. A :latest image built after 0.7.6 carrying 0.7.2 metadata suggests the publish pipeline stamps version from a stale source (old tag/env) rather than the built tree. Consequence: pull-and-restart on dhcp1 (2026-08-13) looked successful but still served 0.7.2. Atimeless_build_infometric (timeless_stack#1) would make this visible on a dashboard; fixing the stamping makes it not happen.Data-plane auth default caused silent ingest failure in dev
The dev instance's auth-on-by-default produced continuous
HTTP 401 data_plane_authorization / missing_credentialsfrom ddnet's log shipper — logs silently dropped rather than queued (client-side half tracked as ddnet#1). Known and being handled separately; recorded here because "default config silently loses data in the simplest local setup" is the kind of first-touch experience that costs early adopters.Error taxonomy: positive note
{"error":"unsupported_capability","reason":"..."}with specific messages ("logical operator "or" is not implemented yet") is genuinely good DX — honest about limits, instantly actionable. The one inconsistency found (uppercase OR bypassing that honesty) is timeless_logs#3.🤖 Generated with Claude Code