From 16e7940b2fe6dfde9f6499992c62057ac9091ef4 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 16 Oct 2026 22:27:15 +0000 Subject: [PATCH 1/8] grafana: qualify the last unqualified "told apart by node" #273 corrected the alert comments to say that node is the per-box label by design but that no node value names a machine today. The Scrapes panel's description on the Regions dashboard still carried the unqualified form, "Two boxes in one stage and region share service_name and are told apart by node", which is the claim #273 removed from alerts.tf -- and it is user-visible, where the comments are not. `git grep "told apart by"` over terraform/ and docs/ now returns only the qualified copy at alerts.tf:1038. Also drops the "per-region today" formulation that survived in the node label rule's comment, which is untrue of dev, whose node is plain "dev". Say what it has in common with the appliance's instead: it does not name a machine. Rewrapped the paragraph it sits in. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AA2iak2zmX9y6XbwCg27Ht --- terraform/envs/grafana/alerts.tf | 12 ++++++------ terraform/envs/grafana/dashboards/forge-regions.json | 2 +- 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/terraform/envs/grafana/alerts.tf b/terraform/envs/grafana/alerts.tf index 5fa94290..e73a2e99 100644 --- a/terraform/envs/grafana/alerts.tf +++ b/terraform/envs/grafana/alerts.tf @@ -1609,12 +1609,12 @@ resource "grafana_rule_group" "appliance" { # scrapes set it: the host and Caddy each through a relabel component, and # the cAdvisor scrape through the static rules its own relabel component # adds, since `up` carries none of the Compose labels the rest of those - # rules read. That node's value is per-region today is a different problem - # from the label going missing, and not the one this rule is about. The - # labelling is a convention living in a config outside these repositories, - # and conventions in prose get dropped when the config is re-implemented -- - # which it will be, at the next box and again if the appliance ever runs its - # own Alloy. + # rules read. That node's value does not name a machine is a different + # problem from the label going missing, and not the one this rule is about. + # The labelling is a convention living in a config outside these + # repositories, and conventions in prose get dropped when the config is + # re-implemented -- which it will be, at the next box and again if the + # appliance ever runs its own Alloy. # # So this fires while the convention is broken, and goes quiet when it is # kept. Prometheus reads an absent label as empty, so node="" selects exactly diff --git a/terraform/envs/grafana/dashboards/forge-regions.json b/terraform/envs/grafana/dashboards/forge-regions.json index 68a0bf3f..56b234c5 100644 --- a/terraform/envs/grafana/dashboards/forge-regions.json +++ b/terraform/envs/grafana/dashboards/forge-regions.json @@ -1718,7 +1718,7 @@ "spec": { "id": 3, "title": "Scrapes", - "description": "Whether Alloy's scrapes for this region answered: the host exporter, and Caddy. It says whether to believe the panels built on those two -- the host metrics and the reconcile age come from the first, the request and error panels from the second -- and when one goes dark its panels read no data rather than zero, which looks much like zero at a glance. It does not cover the whole tab: Ingot's own metrics arrive by a route with no up of its own, so those panels can go blank with both rows here still green. Containers have no scrape either and are in the panel below. A scrape that is attempted and fails reads down; if Alloy itself stops, no up series is produced at all and the row goes blank rather than red, which is why the rule for a node that has gone away tests for absence instead. Two boxes in one stage and region share service_name and are told apart by node.", + "description": "Whether Alloy's scrapes for this region answered: the host exporter, and Caddy. It says whether to believe the panels built on those two -- the host metrics and the reconcile age come from the first, the request and error panels from the second -- and when one goes dark its panels read no data rather than zero, which looks much like zero at a glance. It does not cover the whole tab: Ingot's own metrics arrive by a route with no up of its own, so those panels can go blank with both rows here still green. Containers have no scrape either and are in the panel below. A scrape that is attempted and fails reads down; if Alloy itself stops, no up series is produced at all and the row goes blank rather than red, which is why the rule for a node that has gone away tests for absence instead. Two boxes in one stage and region share service_name; node is the label meant to tell them apart, but every node value today names a stage and at most a region, so nothing here would separate them.", "links": [], "data": { "kind": "QueryGroup", From 4f0224d202628bf23244ce20d35c844d34d89261 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 16 Oct 2026 22:33:56 +0000 Subject: [PATCH 2/8] grafana: unwind the noun clause in the node-label rule's comment "That node's value does not name a machine is a different problem" puts a noun phrase immediately before the main verb, so it garden-paths on first read. Same shape as the phrasing it replaced, so not a regression, but cheap to fix while the line is being touched: state it, then comment on it. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AA2iak2zmX9y6XbwCg27Ht --- terraform/envs/grafana/alerts.tf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/terraform/envs/grafana/alerts.tf b/terraform/envs/grafana/alerts.tf index e73a2e99..4f178bd9 100644 --- a/terraform/envs/grafana/alerts.tf +++ b/terraform/envs/grafana/alerts.tf @@ -1609,7 +1609,7 @@ resource "grafana_rule_group" "appliance" { # scrapes set it: the host and Caddy each through a relabel component, and # the cAdvisor scrape through the static rules its own relabel component # adds, since `up` carries none of the Compose labels the rest of those - # rules read. That node's value does not name a machine is a different + # rules read. No node value names a machine today, but that is a separate # problem from the label going missing, and not the one this rule is about. # The labelling is a convention living in a config outside these # repositories, and conventions in prose get dropped when the config is From 8851c5ed7b96469dbc71bc1f90a3b29a732cc5f2 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 23 Oct 2026 19:17:01 +0000 Subject: [PATCH 3/8] grafana: keep the Scrapes panel neutral on which label names a box Which label should be per-box is an open question with the team: the infra-nodes design doc and runbook both say node is the fleet's per-box identity, but the person migrating the nodes treats it as a choice still to be made between node and appliance. A dashboard tooltip is the wrong place to settle that. State what the panel can and cannot show instead: no label here separates two boxes, because every node value today names a stage and at most a region. Shorter, and true whichever way the decision goes. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AA2iak2zmX9y6XbwCg27Ht --- terraform/envs/grafana/dashboards/forge-regions.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/terraform/envs/grafana/dashboards/forge-regions.json b/terraform/envs/grafana/dashboards/forge-regions.json index 623b793d..337a6b24 100644 --- a/terraform/envs/grafana/dashboards/forge-regions.json +++ b/terraform/envs/grafana/dashboards/forge-regions.json @@ -1718,7 +1718,7 @@ "spec": { "id": 3, "title": "Scrapes", - "description": "Whether Alloy's scrapes for this region answered: the host exporter, and Caddy. It says whether to believe the panels built on those two -- the host metrics and the reconcile age come from the first, the request and error panels from the second -- and when one goes dark its panels read no data rather than zero, which looks much like zero at a glance. It does not cover the whole tab: Ingot's own metrics arrive by a route with no up of its own, so those panels can go blank with both rows here still green. Containers have no scrape either and are in the panel below. A scrape that is attempted and fails reads down; if Alloy itself stops, no up series is produced at all and the row goes blank rather than red, which is why the rule for a node that has gone away tests for absence instead. Two boxes in one stage and region share service_name; node is the label meant to tell them apart, but every node value today names a stage and at most a region, so nothing here would separate them.", + "description": "Whether Alloy's scrapes for this region answered: the host exporter, and Caddy. It says whether to believe the panels built on those two -- the host metrics and the reconcile age come from the first, the request and error panels from the second -- and when one goes dark its panels read no data rather than zero, which looks much like zero at a glance. It does not cover the whole tab: Ingot's own metrics arrive by a route with no up of its own, so those panels can go blank with both rows here still green. Containers have no scrape either and are in the panel below. A scrape that is attempted and fails reads down; if Alloy itself stops, no up series is produced at all and the row goes blank rather than red, which is why the rule for a node that has gone away tests for absence instead. Two boxes in one stage and region share service_name, and no label here would separate them: every node value today names a stage and at most a region.", "links": [], "data": { "kind": "QueryGroup", From e8ec508c988487c77f008587883645ef3717e01f Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 23 Oct 2026 19:20:02 +0000 Subject: [PATCH 4/8] grafana: stop asserting which label is meant to name a box Follows the Scrapes panel. The comments said node is the per-box label and only its value falls short, citing infra-nodes' design doc and runbook. Those do say it, but the choice between node and appliance is still being settled with the team, and these rules work the same either way: they group by all three, so whichever one gains a per-box value splits the groups with no edit here. So the comments now say what the rules cannot see -- no label separates two boxes in a region today -- without naming the label that will fix it. Four comment edits plus the node-label rule's description, which an on-call reads and which called node "the label every per-box rule needs". Co-Authored-By: Petra Jaros Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AA2iak2zmX9y6XbwCg27Ht --- terraform/envs/grafana/alerts.tf | 32 +++++++++++++++----------------- 1 file changed, 15 insertions(+), 17 deletions(-) diff --git a/terraform/envs/grafana/alerts.tf b/terraform/envs/grafana/alerts.tf index a50ec9c5..b16026a0 100644 --- a/terraform/envs/grafana/alerts.tf +++ b/terraform/envs/grafana/alerts.tf @@ -1000,8 +1000,8 @@ resource "grafana_rule_group" "appliance" { # rather than chosen here (FIL-1209). # # Grouped by node as well as region: two boxes in one stage and region share - # service_name and are told apart by node, once the second is given a node - # value of its own. An instant query, so no reduce stage. + # service_name, and nothing here distinguishes them today. node is the + # finest grouping these rules carry. An instant query, so no reduce stage. # # for = 15m is chosen, not specified: a filesystem crossing 40% is not an event # that needs a one-minute response. @@ -1568,8 +1568,8 @@ resource "grafana_rule_group" "appliance" { # The labelling convention, alerted on rather than written down. # - # Every per-box rule in this file needs `node`, because `appliance` is - # - and cannot tell two boxes in one region apart. All three + # Every rule in this file that groups finer than the region needs `node`, + # since `appliance` is - and goes no finer. All three # scrapes set it: the host and Caddy each through a relabel component, and # the cAdvisor scrape through the static rules its own relabel component # adds, since `up` carries none of the Compose labels the rest of those @@ -1601,7 +1601,7 @@ resource "grafana_rule_group" "appliance" { annotations = { summary = "{{ $labels.appliance }} is shipping telemetry with no node label" - description = "A scrape on this appliance is arriving without the node label every per-box rule needs. While that is true, anything reading these series can only work at region granularity, so a second box in the region would be invisible behind the first. Set node and region on the scrape in the host's Alloy config, as the staging section of infra-nodes docs/RUNBOOK.md specifies." + description = "A scrape on this appliance is arriving without the node label these rules group by. While that is true, anything reading these series can only work at region granularity, so a second box in the region would be invisible behind the first. Set node and region on the scrape in the host's Alloy config, as the staging section of infra-nodes docs/RUNBOOK.md specifies." runbook_url = "https://github.com/fil-forge/infra-nodes/blob/main/docs/RUNBOOK.md#when-something-is-wrong" } @@ -2937,20 +2937,18 @@ resource "grafana_rule_group" "appliance_containers" { # region is in the grouping because the dashboard link below reads # $labels.region, and an annotation can only read a label the query emits; # without it the link carried the literal . node is in it because - # the rule beside this one groups by it, and because the grouping starts - # separating boxes the day a node value is per-box, with no edit here. + # the rule beside this one groups by it, and because the grouping separates + # boxes as soon as any of the three carries a per-box value, with no edit + # here. # Neither splits a group, so neither changes when this fires. # - # None of the three tells two boxes in one region apart today, and node is - # the label meant to: infra-nodes' design has node identify the box, and - # names it as what tells apart two nodes sharing a stage, a region and a - # service_name. Only its value falls short: it is the node's name, which - # distinguishes stages and regions, not boxes. A second box in a region - # needs a node value of its own, not a new label -- and until it has one, - # the `unless` in the second branch below would let a live sibling hold the - # 5m side populated and mask a dead cAdvisor. The first branch still catches - # a target that fails rather than vanishes: `up == bool 0` maxes to 1 across - # the pair. + # None of the three tells two boxes in one region apart today: every value + # names a stage and at most a region. Which label should name a box is + # still being settled, so this states the gap rather than anticipating the + # fix. Until one of them carries a per-box value, the `unless` in the + # second branch below would let a live sibling hold the 5m side populated + # and mask a dead cAdvisor. The first branch still catches a target that + # fails rather than vanishes: `up == bool 0` maxes to 1 across the pair. # # The gate stays joined on appliance. Moving it to node would mean `on # (node)` and `max by (node)` together, as the rule beside this one does; From bcf96f45f3cdcc9e21ef42e2c20da7491908408d Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 23 Oct 2026 19:27:32 +0000 Subject: [PATCH 5/8] grafana: drop an over-claim introduced while removing over-claims Round 2. The neutral rewrite said "node is the finest grouping these rules carry", which is false three ways: the rule it annotates also groups by mountpoint; nine rules group by instance, which is /piri-; and on dev appliance is dev-us-east-9 while node is plain dev, so appliance is the finer of the two there. Replaced with the forward-looking reason the container rule already gives -- grouping by all three means the groups split as soon as one carries a per-box value. "Every rule that groups finer than the region needs node, since appliance goes no finer" implied the same ordering and the line below it denies it. Say what actually follows from a missing label instead: the scrape lands in a group of its own and joins to nothing that has it. "Every value names a stage and at most a region" was offered as proof that none of the three separates two boxes, but region's value names a region and no stage, so the proof did not cover one of them. Co-Authored-By: Petra Jaros Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AA2iak2zmX9y6XbwCg27Ht --- terraform/envs/grafana/alerts.tf | 24 +++++++++++++----------- 1 file changed, 13 insertions(+), 11 deletions(-) diff --git a/terraform/envs/grafana/alerts.tf b/terraform/envs/grafana/alerts.tf index b16026a0..6c834281 100644 --- a/terraform/envs/grafana/alerts.tf +++ b/terraform/envs/grafana/alerts.tf @@ -1000,8 +1000,9 @@ resource "grafana_rule_group" "appliance" { # rather than chosen here (FIL-1209). # # Grouped by node as well as region: two boxes in one stage and region share - # service_name, and nothing here distinguishes them today. node is the - # finest grouping these rules carry. An instant query, so no reduce stage. + # service_name, and no label separates them today. Grouping by all three + # means the groups split as soon as one of them carries a per-box value, + # with no edit here. An instant query, so no reduce stage. # # for = 15m is chosen, not specified: a filesystem crossing 40% is not an event # that needs a one-minute response. @@ -1568,8 +1569,8 @@ resource "grafana_rule_group" "appliance" { # The labelling convention, alerted on rather than written down. # - # Every rule in this file that groups finer than the region needs `node`, - # since `appliance` is - and goes no finer. All three + # Every appliance rule groups by `node`, so a scrape arriving without it + # lands in a group of its own and joins to nothing that has it. All three # scrapes set it: the host and Caddy each through a relabel component, and # the cAdvisor scrape through the static rules its own relabel component # adds, since `up` carries none of the Compose labels the rest of those @@ -2942,13 +2943,14 @@ resource "grafana_rule_group" "appliance_containers" { # here. # Neither splits a group, so neither changes when this fires. # - # None of the three tells two boxes in one region apart today: every value - # names a stage and at most a region. Which label should name a box is - # still being settled, so this states the gap rather than anticipating the - # fix. Until one of them carries a per-box value, the `unless` in the - # second branch below would let a live sibling hold the 5m side populated - # and mask a dead cAdvisor. The first branch still catches a target that - # fails rather than vanishes: `up == bool 0` maxes to 1 across the pair. + # None of the three tells two boxes in one region apart today: none of + # their values names more than a stage and a region. Which label should + # name a box is still being settled, so this states the gap rather than + # anticipating the fix. Until one of them carries a per-box value, the + # `unless` in the second branch below would let a live sibling hold the + # 5m side populated and mask a dead cAdvisor. The first branch still + # catches a target that fails rather than vanishes: `up == bool 0` maxes + # to 1 across the pair. # # The gate stays joined on appliance. Moving it to node would mean `on # (node)` and `max by (node)` together, as the rule beside this one does; From eb038e8481a2524e2786792fc5ccaaf0283e6213 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 23 Oct 2026 19:36:08 +0000 Subject: [PATCH 6/8] grafana: anchor two claims the rules themselves disprove Round 3. "Every appliance rule groups by node" was false of the rule the comment annotates: the node-label rule groups by (appliance, job, service_name) and selects node="" on purpose. Every other appliance rule does carry node in its outer grouping -- checked across all three appliance rule groups -- so the fix is "every other", plus a clause saying why this one is the exception, which the comment should have said anyway. "Grouping by all three" had no antecedent: the sentence before it names node, region and service_name, while the rule groups by appliance, region, node and mountpoint. Named them instead of counting them. The node-label rule's description had the same loose referent, and the on-call reading it is looking at the one appliance rule that does not group by node. Now "the other appliance rules group by". Both paragraphs rewrapped whole rather than patched, after two rounds in which a piecemeal rewrap left ragged lines or dropped a word. Co-Authored-By: Petra Jaros Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AA2iak2zmX9y6XbwCg27Ht --- terraform/envs/grafana/alerts.tf | 28 +++++++++++++++------------- 1 file changed, 15 insertions(+), 13 deletions(-) diff --git a/terraform/envs/grafana/alerts.tf b/terraform/envs/grafana/alerts.tf index 6c834281..ebca8d3e 100644 --- a/terraform/envs/grafana/alerts.tf +++ b/terraform/envs/grafana/alerts.tf @@ -1000,9 +1000,10 @@ resource "grafana_rule_group" "appliance" { # rather than chosen here (FIL-1209). # # Grouped by node as well as region: two boxes in one stage and region share - # service_name, and no label separates them today. Grouping by all three - # means the groups split as soon as one of them carries a per-box value, - # with no edit here. An instant query, so no reduce stage. + # service_name, and no label separates them today. Grouping by appliance, + # region and node together means the groups split as soon as one of them + # carries a per-box value, with no edit here. An instant query, so no reduce + # stage. # # for = 15m is chosen, not specified: a filesystem crossing 40% is not an event # that needs a one-minute response. @@ -1569,17 +1570,18 @@ resource "grafana_rule_group" "appliance" { # The labelling convention, alerted on rather than written down. # - # Every appliance rule groups by `node`, so a scrape arriving without it - # lands in a group of its own and joins to nothing that has it. All three + # Every other appliance rule groups by `node`, so a scrape arriving without + # it lands in a group of its own and joins to nothing that has it. This rule + # is the exception: it selects node="" to find those scrapes. All three # scrapes set it: the host and Caddy each through a relabel component, and # the cAdvisor scrape through the static rules its own relabel component - # adds, since `up` carries none of the Compose labels the rest of those - # rules read. No node value names a machine today, but that is a separate - # problem from the label going missing, and not the one this rule is about. - # The labelling is a convention living in a config outside these - # repositories, and conventions in prose get dropped when the config is - # re-implemented -- which it will be, at the next box and again if the - # appliance ever runs its own Alloy. + # adds, since `up` carries none of the Compose labels the rest of those rules + # read. No node value names a machine today, but that is a separate problem + # from the label going missing, and not the one this rule is about. The + # labelling is a convention living in a config outside these repositories, + # and conventions in prose get dropped when the config is re-implemented -- + # which it will be, at the next box and again if the appliance ever runs its + # own Alloy. # # So this fires while the convention is broken, and goes quiet when it is # kept. Prometheus reads an absent label as empty, so node="" selects exactly @@ -1602,7 +1604,7 @@ resource "grafana_rule_group" "appliance" { annotations = { summary = "{{ $labels.appliance }} is shipping telemetry with no node label" - description = "A scrape on this appliance is arriving without the node label these rules group by. While that is true, anything reading these series can only work at region granularity, so a second box in the region would be invisible behind the first. Set node and region on the scrape in the host's Alloy config, as the staging section of infra-nodes docs/RUNBOOK.md specifies." + description = "A scrape on this appliance is arriving without the node label the other appliance rules group by. While that is true, anything reading these series can only work at region granularity, so a second box in the region would be invisible behind the first. Set node and region on the scrape in the host's Alloy config, as the staging section of infra-nodes docs/RUNBOOK.md specifies." runbook_url = "https://github.com/fil-forge/infra-nodes/blob/main/docs/RUNBOOK.md#when-something-is-wrong" } From c0fc254e7979cde8dc19dad2ca8f6a76bf65589c Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 23 Oct 2026 19:45:31 +0000 Subject: [PATCH 7/8] grafana: match each claim's proof to what the claim covers Round 4, no blocking findings. "A scrape arriving without node lands in a group of its own" overstated it: every node-less scrape on an appliance lands in the same node="" group, not one group each. The Scrapes panel said no label there separates two boxes and then offered a fact about node as the proof, which covers one label rather than all of them. Name node as the only label those rows are told apart by, then give the fact about its values. Co-Authored-By: Petra Jaros Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AA2iak2zmX9y6XbwCg27Ht --- terraform/envs/grafana/alerts.tf | 22 +++++++++---------- .../grafana/dashboards/forge-regions.json | 2 +- 2 files changed, 12 insertions(+), 12 deletions(-) diff --git a/terraform/envs/grafana/alerts.tf b/terraform/envs/grafana/alerts.tf index ebca8d3e..af44fdd5 100644 --- a/terraform/envs/grafana/alerts.tf +++ b/terraform/envs/grafana/alerts.tf @@ -1571,17 +1571,17 @@ resource "grafana_rule_group" "appliance" { # The labelling convention, alerted on rather than written down. # # Every other appliance rule groups by `node`, so a scrape arriving without - # it lands in a group of its own and joins to nothing that has it. This rule - # is the exception: it selects node="" to find those scrapes. All three - # scrapes set it: the host and Caddy each through a relabel component, and - # the cAdvisor scrape through the static rules its own relabel component - # adds, since `up` carries none of the Compose labels the rest of those rules - # read. No node value names a machine today, but that is a separate problem - # from the label going missing, and not the one this rule is about. The - # labelling is a convention living in a config outside these repositories, - # and conventions in prose get dropped when the config is re-implemented -- - # which it will be, at the next box and again if the appliance ever runs its - # own Alloy. + # it lands in a group apart from the labelled ones and joins to nothing that + # has it. This rule is the exception: it selects node="" to find those + # scrapes. All three scrapes set it: the host and Caddy each through a + # relabel component, and the cAdvisor scrape through the static rules its own + # relabel component adds, since `up` carries none of the Compose labels the + # rest of those rules read. No node value names a machine today, but that is + # a separate problem from the label going missing, and not the one this rule + # is about. The labelling is a convention living in a config outside these + # repositories, and conventions in prose get dropped when the config is + # re-implemented -- which it will be, at the next box and again if the + # appliance ever runs its own Alloy. # # So this fires while the convention is broken, and goes quiet when it is # kept. Prometheus reads an absent label as empty, so node="" selects exactly diff --git a/terraform/envs/grafana/dashboards/forge-regions.json b/terraform/envs/grafana/dashboards/forge-regions.json index 8e2d2927..bb47cd8f 100644 --- a/terraform/envs/grafana/dashboards/forge-regions.json +++ b/terraform/envs/grafana/dashboards/forge-regions.json @@ -1718,7 +1718,7 @@ "spec": { "id": 3, "title": "Scrapes", - "description": "Whether Alloy's scrapes for this region answered: the host exporter, and Caddy. It says whether to believe the panels built on those two -- the host metrics and the reconcile age come from the first, the request and error panels from the second -- and when one goes dark its panels read no data rather than zero, which looks much like zero at a glance. It does not cover the whole tab: Ingot's own metrics arrive by a route with no up of its own, so those panels can go blank with both rows here still green. Containers have no scrape either and are in the panel below. A scrape that is attempted and fails reads down; if Alloy itself stops, no up series is produced at all and the row goes blank rather than red, which is why the rule for a node that has gone away tests for absence instead. Two boxes in one stage and region share service_name, and no label here would separate them: every node value today names a stage and at most a region.", + "description": "Whether Alloy's scrapes for this region answered: the host exporter, and Caddy. It says whether to believe the panels built on those two -- the host metrics and the reconcile age come from the first, the request and error panels from the second -- and when one goes dark its panels read no data rather than zero, which looks much like zero at a glance. It does not cover the whole tab: Ingot's own metrics arrive by a route with no up of its own, so those panels can go blank with both rows here still green. Containers have no scrape either and are in the panel below. A scrape that is attempted and fails reads down; if Alloy itself stops, no up series is produced at all and the row goes blank rather than red, which is why the rule for a node that has gone away tests for absence instead. Two boxes in one stage and region share service_name, and no label here would separate them: node is the only label these rows are told apart by, and every node value today names a stage and at most a region.", "links": [], "data": { "kind": "QueryGroup", From 56475af1c923811cef39de8fa2c10e6e1c9795c5 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 23 Oct 2026 19:54:00 +0000 Subject: [PATCH 8/8] grafana: scope two more claims, and fix an orphaned wrap Round 5, no blocking findings. The disk rule said no label separates two boxes without saying over what. Scoped it to those series, and named the reason a reader would reach for: instance does not help either, because Alloy sets it to the node name. The Scrapes panel said node is the only label those rows are told apart by, which reads either as the two boxes' rows, where it is exact, or as every row on the panel, where it is not -- host and Caddy are told apart by service_name. Says "those two rows" now. The container grouping paragraph had an orphaned "here." on its own line from an earlier rewrap. Rewrapped. Co-Authored-By: Petra Jaros Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AA2iak2zmX9y6XbwCg27Ht --- terraform/envs/grafana/alerts.tf | 11 +++++------ terraform/envs/grafana/dashboards/forge-regions.json | 2 +- 2 files changed, 6 insertions(+), 7 deletions(-) diff --git a/terraform/envs/grafana/alerts.tf b/terraform/envs/grafana/alerts.tf index af44fdd5..f4a8b614 100644 --- a/terraform/envs/grafana/alerts.tf +++ b/terraform/envs/grafana/alerts.tf @@ -1000,10 +1000,10 @@ resource "grafana_rule_group" "appliance" { # rather than chosen here (FIL-1209). # # Grouped by node as well as region: two boxes in one stage and region share - # service_name, and no label separates them today. Grouping by appliance, - # region and node together means the groups split as soon as one of them - # carries a per-box value, with no edit here. An instant query, so no reduce - # stage. + # service_name, and no label on those series separates them today -- Alloy + # sets instance to the node name too. Grouping by appliance, region and node + # together means the groups split as soon as one of them carries a per-box + # value, with no edit here. An instant query, so no reduce stage. # # for = 15m is chosen, not specified: a filesystem crossing 40% is not an event # that needs a one-minute response. @@ -2942,8 +2942,7 @@ resource "grafana_rule_group" "appliance_containers" { # without it the link carried the literal . node is in it because # the rule beside this one groups by it, and because the grouping separates # boxes as soon as any of the three carries a per-box value, with no edit - # here. - # Neither splits a group, so neither changes when this fires. + # here. Neither splits a group, so neither changes when this fires. # # None of the three tells two boxes in one region apart today: none of # their values names more than a stage and a region. Which label should diff --git a/terraform/envs/grafana/dashboards/forge-regions.json b/terraform/envs/grafana/dashboards/forge-regions.json index bb47cd8f..b013cf22 100644 --- a/terraform/envs/grafana/dashboards/forge-regions.json +++ b/terraform/envs/grafana/dashboards/forge-regions.json @@ -1718,7 +1718,7 @@ "spec": { "id": 3, "title": "Scrapes", - "description": "Whether Alloy's scrapes for this region answered: the host exporter, and Caddy. It says whether to believe the panels built on those two -- the host metrics and the reconcile age come from the first, the request and error panels from the second -- and when one goes dark its panels read no data rather than zero, which looks much like zero at a glance. It does not cover the whole tab: Ingot's own metrics arrive by a route with no up of its own, so those panels can go blank with both rows here still green. Containers have no scrape either and are in the panel below. A scrape that is attempted and fails reads down; if Alloy itself stops, no up series is produced at all and the row goes blank rather than red, which is why the rule for a node that has gone away tests for absence instead. Two boxes in one stage and region share service_name, and no label here would separate them: node is the only label these rows are told apart by, and every node value today names a stage and at most a region.", + "description": "Whether Alloy's scrapes for this region answered: the host exporter, and Caddy. It says whether to believe the panels built on those two -- the host metrics and the reconcile age come from the first, the request and error panels from the second -- and when one goes dark its panels read no data rather than zero, which looks much like zero at a glance. It does not cover the whole tab: Ingot's own metrics arrive by a route with no up of its own, so those panels can go blank with both rows here still green. Containers have no scrape either and are in the panel below. A scrape that is attempted and fails reads down; if Alloy itself stops, no up series is produced at all and the row goes blank rather than red, which is why the rule for a node that has gone away tests for absence instead. Two boxes in one stage and region share service_name, and no label here would separate them: the only label telling those two rows apart is node, and every node value today names a stage and at most a region.", "links": [], "data": { "kind": "QueryGroup",