Skip to content

Root layout, monitoring alerts, freeze hardening - #34

Merged
owenwahlgren merged 14 commits into
mainfrom
refactor/root-layout
Aug 7, 2026
Merged

owenwahlgren merged 14 commits into
mainfrom
refactor/root-layout

Conversation

@owenwahlgren

@owenwahlgren owenwahlgren commented Aug 6, 2026

Copy link
Copy Markdown
Collaborator

The repository root held six runtime config files, the consensus doc, and the inventory example. They now live where their concern lives. #35 (alert rules) merged into this branch, followed by four freeze-hardening commits. This PR is the last substantive change before the repo freezes as the POC.

What changed

Root layout:

  • chains/default/ owns the shared runtime configuration: genesis-template.json, subnet-config.json, node-config.json, and the three chain-config variants. Resolution for every shipped file is the chain's own chains/<name>/ file, then chains/default/, then the legacy root name. Deployment roots cut before this layout keep working unchanged.
  • default joins oracle and management as a reserved chain name.
  • CONSENSUS-TUNING.md moves to docs/. nodes.ini.example joins the other inventory shapes in examples/.
  • make pack and tools/bundle.sh ship chains/, examples/, and docs/ instead of a root file list; bundle still carries legacy root files when re-cutting an old deployment root.

Monitoring alerts (#35):

  • monitoring/alerts.yml: 9 fleet alert rules. apps/settlement-feed/alerts.yml: 2 app rules. Apps ship alert rules next to their dashboards; the compose file mounts one file per app. No height-stall rule: the EVM produces blocks only on transactions, so NodeBehindPeers covers the harmful case.
  • monitoring/prometheus.yml enters git. A stale .gitignore rule kept it out while the compose file, the bundle script, and playbook 06 all reference it; a fresh clone got a broken monitoring stack. The generated targets.json is ignored instead.
  • Playbook 06 gains an Alerts section. The kit sends no notifications by design; the rules are the trigger source for site-specific automation.

Freeze hardening:

  • rename: the module path drops the remote/ relic (avalanche-benchmark/remote -> avalanche-benchmark, 49 files, mechanical). The pack artifact follows: remote-benchmark.tar.gz -> avalanche-benchmark.tar.gz. Last chance before the production fork inherits the old path.
  • pack: the pack artifact now carries the full monitoring stack (prometheus.yml, alerts, compose, weight exporter, provisioning) plus playbooks/ and docs/. A packed control kit could not run playbook 06 before this.
  • ci: .github/workflows/go.yml runs gofmt, build, and test on every PR. Also gives CodeQL's actions analyzer a workflow to analyze.
  • readme: entry point with the Avalanche logo and a start-here table routing each task to its playbook. Manual body unchanged.
  • playbooks: renumbered to the operator order (provision, rootless install, monitoring, load test, failover drill, validator swap, connected P-chain, install app, multi-chain), cross-references swept.
  • bypass removal: the PCHAIN_API_TOKEN rate-limit bypass (transport shim, .env field, docs) is gone. A public kit must not ship secret-token plumbing; isolated deployments use their own P-chain endpoint. The strict .env parser rejects the field loudly; the bundle sanitizer keeps blanking it in old roots.
  • gitignore: the local planning documents are ignored so they can never enter a commit.

Verification

  • go build ./..., go test ./... (15 packages), and gofmt -l clean after every commit, including the module rename.
  • Old-layout compatibility pinned by the existing tests, which build legacy roots.
  • Alert rules: promtool check rules passes (11 rules); deployed to the live Prometheus on the control host, all rules evaluate health=ok.

Open questions

None.

chains/default/ now owns the shared runtime configuration:
genesis-template.json, subnet-config.json, node-config.json, and the
three chain-config variants. Resolution for every shipped file is the
chain's own chains/<name>/ file, then chains/default/, then the legacy
root name, so deployment roots cut before this layout keep working.
default joins oracle and management as a reserved chain name.

CONSENSUS-TUNING.md moves to docs/. nodes.ini.example joins the other
inventory shapes in examples/. pack and bundle ship chains/, examples/,
and docs/ instead of a root file list.
Prometheus evaluates monitoring/alerts.yml (fleet rules) plus one
alerts.yml per installed app, mounted by the compose file. The rules
are the trigger source for operations automation; the kit sends no
notifications. No height-stall rule: the EVM produces blocks only
when transactions arrive, so NodeBehindPeers covers the harmful case.

A stale .gitignore rule from the generated-monitoring era kept
monitoring/prometheus.yml out of git while the compose file, the
bundle script, and playbook 06 all reference it. A fresh clone got a
broken stack: Docker turns the missing file mount into a directory.
The file is shipped now and the generated targets.json is ignored
instead.

All 11 rules pass promtool and evaluate healthy on the live fleet.
Monitoring: Prometheus alert rules, prometheus.yml enters git
@owenwahlgren owenwahlgren changed the title Root layout: chains/default owns the shared runtime configuration Root layout: chains/default owns shared config; monitoring alert rules Aug 6, 2026
The root module was still named avalanche-benchmark/remote from the era
when the Go code lived under remote/. The pack artifact carried the same
relic (remote-benchmark.tar.gz). Mechanical rename, no logic change:
module github.com/ava-labs/avalanche-benchmark, artifact
avalanche-benchmark.tar.gz. This is the last chance before the
production fork inherits the old path.

go build ./..., go test ./..., gofmt all clean.
The pack artifact carried only the Grafana datasource and the
dashboards. A packed control kit could not run playbook 06: no
prometheus.yml, no alert rules, no compose file, no weight exporter, no
provisioning config. The kit now carries the complete monitoring
directory contents plus playbooks/ and docs/, so the control machine is
self-documenting. Still no sources, still no secrets.
The repo had no Go CI; only Socket and CodeQL ran. The frozen POC needs
a tripwire: any future PR gets gofmt, go build, and go test on
ubuntu-latest. This also gives CodeQL's actions analyzer a workflow
file to analyze, which removes one of its two no-source-found
failures.
The manual began mid-thought. It now opens with the Avalanche logo
(docs/avalanche-logo.png, from ava-labs/avalanchego resources), a
one-line pitch, and a start-here table that routes each task to its
playbook. The manual body is unchanged.
@owenwahlgren owenwahlgren changed the title Root layout: chains/default owns shared config; monitoring alert rules Root layout, monitoring alerts, freeze hardening Aug 7, 2026
Comment thread .github/workflows/go.yml Fixed
The numbers encoded writing order, not reading order: rootless install
and monitoring accreted at 05 and 06 while sitting mid-journey. New
order: provision, rootless install, monitoring, load test, failover
drill, validator swap, connected P-chain, install app, multi-chain.
Monitoring moves before the drills because the dashboards should be up
before anything breaks. Cross-references sweep: README table, playbook
cross-links, alert-rule annotations.
PCHAIN_API_TOKEN rode every request to the public API host as a query
argument matched by an infra-side rule. A public kit must not ship
secret-token plumbing: an isolated deployment uses its own P-chain
endpoint, and the public API works without a token at kit rates.
Removed: the transport shim (internal/config/bypass.go), the .env
field, the docs. The strict .env parser now rejects the field by name,
so an old root gets a loud, clear error. tools/bundle.sh keeps blanking
the field, because old deployment roots still carry it.
@owenwahlgren
owenwahlgren force-pushed the refactor/root-layout branch from 56f9dce to 773d4dd Compare August 7, 2026 02:29
MULTICHAIN-PLAN.md, PROPOSAL.md, and PACKAGE-SPEC.md live in the
working tree by convention and never belong in a commit. Convention
failed once; the ignore file makes it structural.
make pack writes avalanche-benchmark.tar.gz now; the ignore rule still
named only the old artifact. Both names stay ignored: old roots carry
the old tarball.
fleet_actual_weight carries machine/identity/dc/l1 labels; the
annotation asked for a node label the exporter never emits and rendered
empty.
CodeQL flagged the unrestricted GITHUB_TOKEN. The jobs only read the
repository; contents: read is the whole requirement.
Review finding: subnetConfigPath and chainTemplatePath consulted
chains/default/ before the oracle-specific legacy root names. An old
oracle deployment root without chains/oracle/ would have silently
picked up the generic consensus parameters instead of the oracle's
own. Specificity now orders every layer: the chain's own file, the
chain-specific legacy name, the shared default, the shared legacy
name. Tests pin the ordering in both packages.
@owenwahlgren
owenwahlgren merged commit a3f97b6 into main Aug 7, 2026
5 of 6 checks passed
@owenwahlgren
owenwahlgren deleted the refactor/root-layout branch August 7, 2026 18:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants