Repository navigation
Chart Piri's CommP queue and alert when its claims stall - #291
Merged
Merged
Conversation
Piri now holds each accepted blob for pdp.aggregation.commp.delay before a CommP task may claim it (fil-forge/piri#187). Three panels on the Regions dashboard show the queue per node: blobs held by the delay against blobs past it with no task yet; blobs released before aggregation by whether their CommP had been computed, which is what the delay saves; and, from the age histogram Piri records for those releases, the share released within 1, 5, 15, 30 and 60 minutes of acceptance, which is what a delay of each length catches and so whether the default should move. One appliance rule warns when the oldest due blob has gone unclaimed for ten minutes, which is the task engine down or saturated. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
alanshaw
force-pushed
the
ash/feat/piri-commp-delay-panel
branch
from
October 21, 2026 10:55
6a5e72f to
14e29fb
Compare
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
The focused observability changes match the companion metrics and existing alert patterns, with no unresolved findings.
No open issues 🎉
What changed in this PR
Adds Grafana visibility into Piri’s delayed CommP processing, complementing the metrics introduced in fil-forge/piri#187.
Changes:
- Adds panels for queue depth, early releases, and delay effectiveness.
- Adds a warning for stalled CommP claims, remaining quiet when metrics are absent.
- Documents the new alert.
| File | Description |
|---|---|
| terraform/envs/grafana/dashboards/forge-regions.json | Adds three Piri panels to the region tab. |
| terraform/envs/grafana/alerts.tf | Adds the stalled CommP claims warning. |
| docs/observability.md | Lists the new alert. |
Review effort: Balanced
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Companion to fil-forge/piri#187, which holds each accepted blob for
pdp.aggregation.commp.delay(one hour by default) before a CommP task may claim it, so a blob deleted soon after upload costs no CommP and no on-chain root.Panels (Forge Regions, region tab, below the Ingot panels):
stage.before_commpis what the delay is saving.Alert (appliance group, warning): Piri CommP claims have stalled, when
piri_pdp_commp_due_blobs_oldest_secondshas been over ten minutes for ten minutes. A due blob is claimed on the task engine's next poll, so this is the engine down or saturated, not CommP being slow.The metrics ship once piri#187 is deployed; until then the panels are empty and the rule has no series, which
no_data_state = OKkeeps quiet.tofu validateandtofu fmt -checkpass; the dashboard is normalised.🤖 Generated with Claude Code