[Peeja]

Written by Claude.

This adds an AGENTS.md with one short section, Observability. Claude Code v2.1.277 and later reads AGENTS.md itself when no CLAUDE.md exists (docs), so there is no CLAUDE.md.

When a change adds or alters something that runs on an appliance, the section tells an agent to add what on-call needs in the same PR: logs, metrics, a line in docs/observability.md, and an alert or dashboard panel in fil-forge/infra-central. It also says to skip this for changes nothing operational depends on.

The how-to lives in a new "Adding a signal" section of docs/observability.md, so it isn't loaded into every session:

• Logs:
  • Container output ships on its own.
  • Staging ships only the reconcile unit from the host journal. A new host service there is recorded in the staging section of the runbook, and the PR says that the host's Alloy config, which lives outside this repo, needs the change.
• Metrics, pushed: a service that pushes OTLP sets service.name to its Compose service name and service.namespace="forge", and sends to Alloy on port 4318. That needs no Alloy change on dev or staging, unless the service runs replicas: dev's Alloy keeps a per-replica instance and a shared log service_name only for Piri's, so a new replicated service needs those rules extended, and staging, which sets instance to the node for every series and names logs per Compose service, needs rules of its own.
• Metrics, scraped:
  • A scraped service is relabelled to set service_name (from SERVICE_NAME_PREFIX) and instance.
  • On dev, appliance, node and region come from the writer. On staging only appliance does, so the runbook entry also sets node, region and instance, and the PR says the host's Alloy config needs it.
  • Host-side signals can follow the deploy-stamp pattern.
• Alerts and panels live in fil-forge/infra-central. The PR either opens a companion PR there or says that one is needed.

Companion PRs add the same guidance to fil-forge/piri#174, fil-forge/infra-central#253 and the Forge service repos.

Checks

• make check-shell check-compose passes.
• git diff --check is clean.

✴️

🤖 Generated with Claude Code

https://claude.ai/code/session_01JCF8gyb8jNRmHmWqQmVhRJ

---

Generated by Claude Code

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

[Peeja]

Claude reviewer says… (1st round, 4473c40)

Two should-fix findings and two nits; nothing blocking.

Verified

• Journal shipping: dev ships the whole journal, staging only the reconcile unit (docs/observability.md:16-18).
• Files cited: config.alloy exists, and RUNBOOK's staging section says what the host's Alloy config must contain.
• Deploy stamp: it is deploy_last_success_timestamp, written by stamp_deploy_success in scripts/host/lib.sh.
• infra-central side: the Forge Regions groups and the forge-regions dashboard exist there.
• Checks: make check-shell passes and git diff --check is clean.

Findings

1. should-fix, AGENTS.md:18. The labels list is incomplete.
   • Every appliance series also needs service_name (appliance-<stage>-<region>-<service>; RUNBOOK:220-223). On staging, that label is what keeps a series inside the appliance's queries and rules (RUNBOOK:313-315, observability.md:184-186).
   • On dev, appliance, node and region come from the writer's external_labels, not from each scrape. The host and Caddy scrapes set service_name and instance with a prometheus.relabel.
2. should-fix, AGENTS.md:11-13. "add it to that host's Alloy config" isn't something an agent can do: that config lives outside the repo.
3. nit. "add a scrape" misses the push path. Piri pushes OTLP to the apps receiver.
4. nit. The section lacks piri's "Skip this for changes nothing operational depends on".

✴️

---

Generated by Claude Code

[Peeja]

Claude author says…

Fixed in 7b5c0f4.

1. Labels. The metrics bullet now covers both paths:

   • A pushed OTLP service only needs the apps receiver.
   • A scraped service goes through a prometheus.relabel that sets service_name (appliance-<stage>-<region>-<service>) and instance, as the host and Caddy ones do.

   It says appliance, node and region come from the writer's external_labels, and that on staging service_name keeps a series inside the appliance's queries and rules, so the same requirement goes into the runbook's staging section.

2. Staging host config. For a new host service on staging, the logs bullet now says to add what must ship to the runbook's staging section, and to say in the PR that the host's config needs the change.

3. Push path: covered in 1.

4. Skip line: added.

make check-shell passes and git diff --check is clean. This round had should-fix findings, so round 2 follows.

✴️

---

Generated by Claude Code

[Peeja]

Claude reviewer says… (2nd round, 7b5c0f4)

Round 1's four items are fixed. Two new should-fix findings came from the rewrite; nothing is blocking.

Verified

• Dev OTLP path: the apps receiver exists, and its relabel sets service_name from job and instance from FILONE_NODE.
• Dev scrape path: the host and Caddy relabels build service_name from SERVICE_NAME_PREFIX (appliance-${STAGE}-${REGION_LABEL}).
• Dev labels: both external_labels blocks carry node, region and appliance.
• Staging service_name: it keeps a series in the appliance's queries and rules (observability.md:183-186, RUNBOOK:313-315).
• Checks: make check-shell passes and git diff --check is clean.

Findings

1. should-fix, AGENTS.md:16-17. "Needs only to send it to the apps receiver" holds only on dev, and only when the service sets its resource attributes.
   • On dev, service_name comes from service.namespace/service.name.
   • On staging, services push to the host's Alloy, which keeps only job=~"forge/.+" (RUNBOOK:609-616). A service without service.namespace="forge" is silently dropped there.
2. should-fix, AGENTS.md:20-23. "appliance, node and region come from the writer's external_labels" is dev only. On staging only appliance is on the writer (RUNBOOK:224-230), so a staging entry also needs node, region and instance.
3. nit. The PR body still describes the round 1 file.
4. nit. Spelling out the service_name form invites a hardcoded value. Say how it's built.
5. nit, outside this diff. Dev Piri pushes every 10s (publish_interval), but config.alloy:213 and observability.md:26 say 30 seconds.

✴️

---

Generated by Claude Code

[Peeja]

Claude author says…

Fixed in 92d2a09.

1. OTLP path. A pushed service now "sets service.name to its Compose service name and service.namespace="forge", and sends to Alloy on port 4318": the apps receiver on dev, the host's Alloy on staging. The text adds that "on staging, a series without that namespace is dropped".
2. Staging labels. The text now says that on staging only appliance is on the writer. The runbook entry for a scraped service sets service_name, node, region and instance, as the existing staging rules do.
3. PR body: rewritten to match the file.
4. service_name: now "built as sys.env("SERVICE_NAME_PREFIX") + "-<service>"".
5. Push interval: left out of this PR. It's a mismatch between existing comments and config, so I'll raise it separately.

make check-shell passes and git diff --check is clean. Round 2 had should-fix findings, so round 3 follows.

✴️

---

Generated by Claude Code

[Peeja]

Claude reviewer says… (3rd round, 92d2a09)

Ready to merge: both round 2 should-fix findings are fixed, and five nits remain.

Verified

• Dev OTLP path: the apps receiver listens on 4318. Piri and Ingot point at alloy:4318. job is namespace/name, and dev keeps the name, so service_name matches the logs when service.name equals the Compose service name.
• Staging OTLP path: services push to host.docker.internal:4318. The keep on job=~"forge/.+" drops everything else, and RUNBOOK:461 says a later Forge service needs no change.
• Dev scrape path: the relabels build service_name from SERVICE_NAME_PREFIX and set instance. Both external_labels blocks carry node, region and appliance.
• Staging labels: only appliance is on the writer, and the existing rules set the rest. The RUNBOOK's staging section is where those requirements are recorded.
• Checks: make check-shell passes and git diff --check is clean.

Findings

1. nit. "as Piri and Ingot do" is under Metrics, but docs/observability.md:31 still lists Ingot's application metrics as not shipped. Either the phrase or that line is out of date.
2. nit. The bullet gives the port but not the address for each stage (alloy:4318, host.docker.internal:4318).
3. nit. The staging scrape sentence doesn't say to note in the PR that the host's config needs the change, as the Logs bullet does.
4. nit. The Metrics bullet is one dense paragraph. Sub-bullets would make it easier to act on.
5. nit, PR body. It dropped "to its Compose service name".

✴️

---

Generated by Claude Code

[Peeja]

Claude author says…

Fixed in a44f1e9.

1. "as Piri and Ingot do": now reads "as Piri does". The observability.md:31 line is outside this PR. Ingot's main exports metrics, and infra-central's rules read them, so the line looks stale, but whether the pinned image already does is a question for whoever maintains it. I'm raising it separately.
2. Addresses: both stage addresses are now named, and the bullet points at Ingot's entry in each stage's apps/compose.yml.
3. PR note: the staging scrape now says to "say in the PR that the host's Alloy config needs it".
4. Density: the Metrics bullet is split into pushed, scraped on dev, scraped on staging, and host-side.
5. PR body: the qualifier is restored.

make check-shell passes and git diff --check is clean. This round found only nits, so there won't be another.

✴️

---

Generated by Claude Code

[bajtos]

It's a great idea to add some instructions for coding agents! 👏🏻

I am wondering - how did you evaluate how effective these instructions are?

[Peeja]

Claude says… I didn't measure whether they work; nothing here is an evaluation of their effect. The review rounds on this PR checked that the instructions are accurate: the labels, paths and ports against config.alloy, the compose files and the runbook. They didn't check whether an agent following them actually adds telemetry.

One way to find out, as a suggestion rather than something that's been tried: give an agent a small feature task in this repo, with and without the section, and see whether it adds a metric, the doc line and a note about an infra-central alert. A cheaper signal would be whether the next few agent-authored PRs here mention observability at all.

✴️

---

Generated by Claude Code

[Peeja]

Claude reviewer says… (5th round, 446d7d5)

Approve. Nits only.

Verified

• Dev metrics: prometheus.relabel "apps" (config.alloy:293-333) keeps the reported instance only for forge/piri.
• Dev logs: the log rules (:98-124) give only piri-<n> a shared service_name and an instance.
• Staging: its filone_forge rule (RUNBOOK:607-641) flattens instance.
• The compose addresses and "as Piri does" are right.
• make check-shell check-compose and git diff --check pass.

Findings

1. nit, docs/observability.md:323: "keeps that apart" has no clear referent.
2. nit, :325-326: staging also names logs per Compose service (RUNBOOK:236-242), so the replica caveat should cite that too. At 10 lines, the bullet also reads better split in two.
3. nit, PR body: the "Metrics, pushed" caveat names dev's rules but not staging.

✴️

---

Generated by Claude Code

[Peeja]

Claude author says…

Fixed in 7471654.

1. Now "keeps replicas apart only for Piri".
2. "Replicas are the exception" is a paragraph of its own now, and it names both of staging's reasons: instance set to the node, and logs named per Compose service.
3. The PR body names staging too.

Checks: make check-shell check-compose, git diff --check.

This round found only nits, so there won't be another.

✴️

---

Generated by Claude Code

[Peeja]

@bajtos Yeah, there isn't really a good way I know of to validate this other than trying it out for a while and iterating. I figure this is better than not.

[bajtos]