[backend-scheduler] expose pending job depth and fix job-duration bucketing

[zalegrala]

What this PR does:

Adds the missing scale signal for backend work, and fixes the job-duration histogram so its percentiles mean something.

tempo_backend_scheduler_jobs_pending{tenant, job_type} — jobs enqueued and not yet dispatched. Nothing exposed queue depth before: jobs_created_total counts dispatch and jobs_active counts work already handed to a worker, so a submission that fanned out 145 jobs reported 1 while the other 144 were invisible to Prometheus.

Depth is also the only signal that can drive scale-up. jobs_active is bounded by the worker count and rises only as capacity rises, so reading it to decide whether to add capacity is circular.

This matters most for redaction, which suppresses the tenant's compaction while it runs. An autoscaler driven by compaction backlog therefore watches that backlog disappear exactly when redaction work begins, and scales down mid-run. A trigger on sum(tempo_backend_scheduler_jobs_pending{job_type="JOB_TYPE_REDACTION"}) holds scale for the duration and releases when the queue drains.

The gauge Reset()s before each publish. It is keyed by tenant, and a tenant whose queue drains stops appearing in the snapshot rather than reporting zero — without the reset its last non-zero value would persist for the process lifetime, so a finished redaction would look permanently backlogged and an autoscaler would never scale back down. That is the behaviour the test pins.

backend_scheduler_job_duration_seconds buckets. The client-library defaults stop at 10s and place no boundary between 2.5s and 5s. Fleet-wide over 7 days, ~60% of redaction jobs land in that single bucket, so p50, p90 and p99 are all linear interpolations inside it — they move together and describe nothing. Now powers of two from 10ms to ~11m, which resolves fast retention jobs, splits the redaction mass, and gives long jobs a real bucket instead of +Inf.

Dashboards using histogram_quantile are unaffected. Queries pinned to specific le values need updating.

Native histograms. tempodb_cache_store_size_bytes was the only histogram in the tree without native-histogram configuration; it now matches the other 33, so coverage is complete at 34/34.

Worth noting separately: ops currently stores only the _bucket series for backend_scheduler_job_duration_seconds — no _sum, no _count, and no native histogram — even though the code has emitted native-histogram config for this metric all along. That is an ingestion-path question rather than a code one, and it is why the classic buckets still matter today.

Dashboard. The Backend Work dashboard gains a "Pending Redaction Jobs" panel leading the Redaction row. "Active Redaction Jobs" reads the same on the first block as on the last — it counts what a worker holds right now, which is bounded by the worker count — so nothing on the dashboard previously showed how much of a redaction was left. Both job-duration panels also gain native-histogram queries alongside the classic-bucket ones, so the panel renders on stacks that retain either representation.

Which issue(s) this PR fixes:

N/A (tracked internally).

Checklist

• Tests updated
• Documentation added
• Changelog entry added under .chloggen/

[Copilot]

Pull request overview

This PR improves observability and scaling signals for the backend scheduler by exposing pending queue depth, correcting job-duration histogram bucket boundaries for meaningful percentiles, and extending native-histogram coverage—along with corresponding dashboard, docs, and changelog updates.

Changes:

• Add tempo_backend_scheduler_jobs_pending{tenant,job_type} and publish it on the scheduler maintenance loop from a per-tenant/type queue-depth snapshot.
• Replace backend_scheduler_job_duration_seconds buckets with powers-of-two from 10ms to ~11m and complete native-histogram options for tempodb_cache_store_size_bytes.
• Update Tempo mixin dashboards (and compiled output) plus docs/changelog entries to reflect the new/updated metrics.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
tempodb/backend/cache/cache.go	Adds shared namespace constant and native-histogram opts for the cache store-size histogram.
modules/backendscheduler/work/work.go	Adds PendingJobCounts() snapshot API for pending queue depth.
modules/backendscheduler/work/interface.go	Exposes PendingJobCounts() on the scheduler work interface.
modules/backendscheduler/metrics.go	Adds backend_scheduler_jobs_pending gauge, updates job-duration histogram buckets, and adds publisher helper.
modules/backendscheduler/backendscheduler.go	Publishes pending-depth metric on the maintenance ticker.
modules/backendscheduler/work/pending_counts_test.go	Unit test for correctness and “no stale series” semantics of the pending-depth snapshot.
modules/backendscheduler/metrics_pending_test.go	Unit test ensuring Reset() behavior removes drained-series from jobs_pending.
operations/tempo-mixin/dashboards/tempo-backendwork.json	Adds “Pending Redaction Jobs” panel and adds native-histogram queries to job-duration panels.
operations/tempo-mixin-compiled/dashboards/tempo-backendwork.json	Regenerated compiled dashboard JSON reflecting mixin changes.
docs/sources/tempo/reference-tempo-architecture/components/compaction.md	Documents the new jobs_pending metric in the compaction/backendscheduler metrics list.
.chloggen/backendwork-dashboard-redaction-progress.yaml	Changelog entry for dashboard improvements around redaction progress and native hist queries.
.chloggen/backend-scheduler-jobs-pending.yaml	Changelog entry for new pending-depth metric, bucket changes, and native-hist coverage completion.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.