Add automated templates health check by cxalem · Pull Request #420 · solana-foundation/templates

cxalem · 2026-06-29T13:41:53Z

What this adds

A script, pnpm health, that checks the health of every template in the repo.

Why

CI scaffolds and builds the non-community templates, but the community templates are not tested at all. When a dependency update breaks one, nobody notices until a user runs into it. This script checks every template and reports what is broken.

How it works

Run pnpm health from the repo root. For each template it copies the template into a temporary folder, installs dependencies with that template's own package manager, runs its build or ci script, and collects a few extra signals. It writes a JSON file and a readable Markdown report under health-reports/. It does not change anything in the repo and never touches mainnet or real funds.

This is maintainer tooling that lives at the repo root, so it is not included when someone scaffolds a single template with create-solana-dapp.

What it checks

Build: does the template install and build, or pass its ci script
Rust: does the program compile (cargo check or cargo build)
Dependencies: which packages are outdated
Security: known vulnerabilities from npm audit
Deprecations: direct dependencies the registry marks as deprecated
Docs: commands in the README that do not match the package scripts

A template only counts as failing if it does not build. Outdated or vulnerable dependencies show as warnings, not failures, so this does not duplicate Dependabot. Templates that need setup or credentials are skipped rather than marked as failing.

Where this is going

This is the first step. The longer term goal is an autonomous agent that does not just build each template but runs it and confirms it actually works. Planned next steps, not part of this PR:

Run this on a schedule in CI so breakages are caught automatically.
Add Playwright to load each web template in a browser and confirm the UI renders without errors.
For templates that need a wallet, generate a throwaway keypair, request a devnet airdrop, connect, and run the core action end to end on devnet.

greptile-apps · 2026-06-29T13:49:06Z

Greptile Summary

This PR adds a repo-level health check for templates. The main changes are:

New pnpm health command for install, build, Rust, audit, dependency, deprecation, docs, and boot checks.
JSON and Markdown health reports under health-reports/.
Template discovery and classification helpers for package managers, Rust layouts, and credential needs.
Unit tests for reporting, status handling, semver parsing, deprecation parsing, and secret detection.
Agent docs for running health checks and future UI testing.

Confidence Score: 4/5

This is close, but the health results can still be wrong under parallel runs.

Overlapping health processes can delete each other's active temp directories.
Parallel boot checks can make healthy web templates fail when they compete for the same default port.
The report-writing, baseline, pnpm advisory, Anchor discovery, and credential-detection fixes look sound.

scripts/health/checks.ts

Important Files Changed

Filename	Overview
scripts/health/checks.ts	Adds the core template checks and cleanup logic, with remaining issues in shared temp cleanup and parallel booting.
scripts/health-check.ts	Adds CLI orchestration, retries, baseline diffing, timestamped reports, and write-result handling.
scripts/health/enumerate.ts	Adds template discovery, package-manager detection, Rust manifest detection, and credential-name parsing.

_{Reviews (6): Last reviewed commit: "fix: address health-check review feedbac..." | Re-trigger Greptile}

github-actions · 2026-07-07T00:26:57Z

This pull request has been automatically marked as stale because it has not had any activity for 7 days.

It will be closed in 7 days if no further activity occurs.

If you believe this PR is still relevant, please add a comment or push new commits to keep it open.

Thank you for your contributions!

Adds `pnpm health`, a script that checks every template and reports across five dimensions: build (install plus each template's ci or build script), Rust compile (cargo check or build, with --cargo-test for the full suite), dependency freshness, npm audit, deprecated direct deps, and doc drift. Only build, Rust, and boot can mark a template "fail". Outdated deps, audit advisories, and deprecations cap at "warn" so the tool does not duplicate dependabot or flag a working template as broken. Templates that need setup or credentials are marked "skip", and functional failures are retried once in isolation to tell flaky apart from real. Each template installs in an isolated temp dir using its pinned package manager, and temp dirs are swept on exit so artifacts never pile up. Output is a JSON artifact plus a Markdown report under health-reports (gitignored). Adds unit tests via `pnpm health:test` with no new dependencies.

health-check.md covers the judgment layer the script does not decide: runtime correctness on devnet, whether a whole stack is still current, and how to read the report. ui-testing-plan.md records the agreed path for browser-level testing, which is the fidelity ladder, the headless wallet-standard approach, and the boot to Playwright seam, so future work starts from a clear plan. AGENTS.md links both docs and the new commands.

A dev server that boots but serves error pages was counted as healthy because any HTTP response passed the boot check. Now only successful statuses pass. Error statuses keep polling until the 60s deadline since dev servers can return 500 while still compiling, and if the server never recovers the check fails with the last status and server output recorded.

Tools like Next.js hard-code ANSI color sequences into their error output even with FORCE_COLOR=0 set, so build failure notes in the Markdown report rendered as unreadable escape-code soup. Strips CSI and OSC sequences in tail(), the single point every human-facing note flows through, and sets NO_COLOR=1 on spawned commands so well-behaved tools skip colors entirely.

The repo's own scripts were never type-checked: tsx strips types without checking and lint only ran the structure check plus prettier. Adds tsc --noEmit to pnpm lint, which validate-templates already runs on every PR, so CI picks it up with no workflow changes. Fixes the existing errors: a nullable packageManager reaching run() and spawn() in the health checks (now a ResolvedRunOptions type), result narrowing when report writes fail, four unused imports, and an ArrayBufferLike assertion in image-utils. Also declares react and @types/react at the root, which image-utils imported without any package declaring it.

greptile-apps · 2026-07-13T16:45:45Z

+  const deadline = Date.now() + 60_000
+  const kill = () => {
+    try {
+      child.kill('SIGKILL')


Dev server survives cleanup

This only kills the direct package-manager process from run dev. The shell and the actual next dev or vite child can keep running after the boot check returns, so later template checks can see ports held by an older template server. When that stale server responds successfully, the boot result can be based on the wrong template instead of the one currently being checked. Please terminate the full process tree, for example by starting the dev command in its own process group and killing that group in finally.

dev-jodee

Some couple of potential issues + we need a rename of all single letter variables, functions, etc. :)

dev-jodee · 2026-07-13T17:21:43Z

+  const r = await run(pm, args, workDir, 120_000)
+  let parsed: Record<string, { current?: string; latest?: string }>
+  try {
+    parsed = JSON.parse(r.output || '{}')


r.output contains both stdout and stderr, so I think this can crash, and then the try catch will hide the issue? (same for audit section below)

dev-jodee · 2026-07-13T17:25:59Z

+    child.stderr.on('data', cap)
+    const timer = setTimeout(() => {
+      timedOut = true
+      child.kill('SIGKILL')


This will only kill the direct process (npm or pnpm or wtv package manager), but if that process spawns other children (like vite serve, cargo, dev servers, etc.) they won't get killed. I think a possible solution is to spawn it detached and then sigkill the process group instead of just the direct child. (same for the kill a couple hundred lines below too

dev-jodee · 2026-07-13T17:30:45Z

+  const envExample = join(dir, '.env.example')
+  if (existsSync(envExample)) {
+    const body = readFileSync(envExample, 'utf-8').toLowerCase()
+    if (SECRET_HINTS.some((h) => body.includes(h.replace(' ', '_')) || body.includes(h))) {


this will "skip" if i.e. a .env.example file has secret in a comment (instead of an actual env missing), think we'd need to parse line by line and check for actual _KEY= or something

dev-jodee · 2026-07-13T17:37:04Z

+import type { OutdatedDep } from './types.js'
+
+/** Parse a version like "^1.2.3" / ">=2.0.0" / "1.2.3-rc.1" into [major, minor, patch]. */
+export const parseVersion = (v: string): [number, number, number] | null => {


in general across the pr, we shouldn't have var names that are 1 letter, if v here is version then it should be named version, same for c, l, etc. below it makes the code hard to read and doesn't really give us any advantages

Kills the full process tree of spawned commands (detached process groups) so orphaned dev servers can't hold ports and answer a later template's boot probe. Parses JSON from stdout only since stderr warnings corrupted the parse. Detects credential requirements from .env.example variable names instead of raw text so comments can't cause a false skip, with unit tests. Renames all single letter variables per review.

greptile-apps · 2026-07-13T22:14:32Z

+export const sweepOrphanTempDirs = (): number => {
+  let removed = 0
+  try {
+    for (const name of readdirSync(tmpdir())) {
+      if (name.startsWith(TEMP_PREFIX)) {
+        rmSync(join(tmpdir(), name), { recursive: true, force: true })
+        removed++
+      }
+    }
+  } catch {
+    /* tmpdir not readable - nothing to sweep */
+  }
+  return removed
+}


Temp sweep races

This sweep only narrows the prefix, but it still deletes every matching temp directory in the shared system temp folder. If two pnpm health runs overlap on the same runner, the later run removes the first run's active template copies because both use sf-templates-health-. The first run can then keep installing or building from a deleted working directory and report templates as broken when they are not. Use a per-run parent directory, a lock, or another ownership marker so startup cleanup cannot remove active work from another process.

greptile-apps · 2026-07-13T22:14:33Z

+  const start = Date.now()
+  // detached: the dev command becomes its own process-group leader, so killProcessTree in
+  // the finally below takes out the whole tree (pm -> next/vite -> workers), not just the pm.
+  const child = spawn(opts.packageManager, ['run', 'dev'], {
+    cwd: workDir,
+    env: { ...process.env, FORCE_COLOR: '0', NO_COLOR: '1' },
+    detached: true,


Boot ports collide

pnpm health --boot still checks templates in parallel, but each dev script keeps its default port. Two web templates can start at the same time and both try to bind the same Next or Vite port. The second server exits with EADDRINUSE, and this path reports that template as a boot failure even though it would boot by itself. Assign an isolated port per template or run boot checks serially so parallel health runs do not create false failures.

cxalem requested a review from catmcgee as a code owner June 29, 2026 13:41

greptile-apps Bot reviewed Jun 29, 2026

View reviewed changes

Comment thread scripts/health/checks.ts Outdated

github-actions Bot added the stale label Jul 7, 2026

cxalem force-pushed the feat/templates-health-check branch from fc8679a to 3933d5c Compare July 10, 2026 18:00

github-actions Bot removed the stale label Jul 11, 2026

cxalem added 4 commits July 13, 2026 16:59

fix: address health-check review feedback

14b48e0

cxalem force-pushed the feat/templates-health-check branch from 3933d5c to 13fe8f9 Compare July 13, 2026 15:01

cxalem added 2 commits July 13, 2026 18:38

greptile-apps Bot reviewed Jul 13, 2026

View reviewed changes

dev-jodee requested changes Jul 13, 2026

View reviewed changes

greptile-apps Bot reviewed Jul 13, 2026

View reviewed changes

Uh oh!

Conversation

cxalem commented Jun 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this adds

Why

How it works

What it checks

Where this is going

Uh oh!

greptile-apps Bot commented Jun 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 4/5

Important Files Changed

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jul 7, 2026

Uh oh!

greptile-apps Bot Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

dev-jodee left a comment

Choose a reason for hiding this comment

Uh oh!

dev-jodee Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

dev-jodee Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

dev-jodee Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

dev-jodee Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

cxalem commented Jun 29, 2026 •

edited

Loading

greptile-apps Bot commented Jun 29, 2026 •

edited

Loading