Feature: embedded recording rules and alerts

[awksedgreep]

Summary

Add a deliberately small embedded recording-rule and alert evaluator over existing bounded Timeless metric, log, and trace query surfaces. Provide durable local detection without attempting to recreate the complete Prometheus/Alertmanager ecosystem.

Candidate capabilities

• Periodic metric recording rules that write derived series through public ingest contracts.
• Metric threshold/range conditions.
• Bounded log-count or exact-filter conditions.
• Bounded slow/error trace conditions.
• Durable inactive, pending, firing, and resolved state.
• Optional webhook and explicitly configured local notification adapters.
• Rule health, missed evaluations, duration, work, errors, and notification counters.

Scheduler and evaluation contract

• One scheduler owner per database/rule set, with canonical database identity.
• Evaluation time, interval alignment, lookback, for duration, missed intervals, restart, and clock-jump behavior are versioned.
• Every query has explicit result/work/state/deadline bounds.
• Rules execute against a defined committed snapshot and never observe partial ingest batches.
• Recording outputs have loop detection and a clear timestamp/provenance contract.
• State and notification transitions are durable and idempotent.

Guardrails

• Begin with a narrow documented expression/condition surface; do not imply full PromQL, MetricsQL, LogsQL, or Alertmanager compatibility.
• Notification secrets are not stored in telemetry rows or exported by health endpoints.
• Slow rules cannot block ingest or starve storage maintenance.
• Delivery semantics (at-most-once, at-least-once, retry/backoff) are explicit per adapter.

Acceptance criteria

• Deterministic-clock tests cover pending/firing/resolved transitions, restart, downtime, delayed evaluation, and clock changes.
• Cancellation, timeouts, conflicting writers, rollback, retention, missing data, NaN/null, and partial signal planes are pinned.
• Scheduler ownership, resource isolation, notification retry, and duplicate prevention survive crash/reopen testing.