Skip to content

Fix example runtime test lanes - #73

Merged
0xjcf merged 5 commits into
betafrom
fas/example-runtime-test-lanes
Jun 29, 2026
Merged

Fix example runtime test lanes#73
0xjcf merged 5 commits into
betafrom
fas/example-runtime-test-lanes

Conversation

@0xjcf

@0xjcf 0xjcf commented Jun 29, 2026

Copy link
Copy Markdown
Owner

Summary

Fixes the Phase B closeout test-lane shape by replacing the one-off smart-home Vitest command in the root test script with explicit package, script, example-runtime, and full test lanes.

What changed

  • Added scripts/test-examples.mjs to discover top-level example roots with runtime tests and run Vitest against each example's own Vite/Vitest config without making examples pnpm workspace members.
  • Added scripts/__tests__/test-examples.test.mjs to guard the discovery list for current runtime-tested examples.
  • Split root scripts into test:packages, test:scripts, test:examples, and test:full; root test now runs package tests only.
  • Pointed FAS testCommand at npm run test:full so FAS verification covers package tests plus example runtime tests.
  • Added CI steps for script tests and example runtime tests.
  • Tracked FAS task state files for the igniteTools follow-up slate.

Why

PR #71 tactically made root npm run test run only the smart-home example test. That caught the changed example test but left the long-term test organization wrong because other top-level examples already have runtime tests and examples intentionally remain outside pnpm-workspace.yaml.

Validation

  • npm run test:examples
  • npm run test:full
  • fas validate-task
  • .fas/scripts/verify.sh --full
  • fas commit-plan check

Final full verification passed before push. The branch push hook also ran package tests successfully.

Summary by CodeRabbit

  • Bug Fixes

    • Improved confidence in releases by ensuring example runtime tests are included in the main validation flow.
    • Updated the default validation lane to use the broader full test command.
  • Tests

    • Added automated discovery and execution coverage for example runtime tests (including fail-fast for invalid layouts).
    • Updated CI to run script tests and example runtime tests in addition to existing checks.
    • Restructured workspace test scripts to support test:packages, test:scripts, test:examples, and test:full.

@coderabbitai

coderabbitai Bot commented Jun 29, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 1f5b5f35-2b96-4446-9fd6-ffd92e8dea5d

📥 Commits

Reviewing files that changed from the base of the PR and between f128eff and d3b52a9.

📒 Files selected for processing (4)
  • .fas/TASKS.md
  • .fas/tasks/build-a-headless-anthropic-tool-use-dogfood-example-for-igni.md
  • scripts/__tests__/test-examples.test.mjs
  • scripts/test-examples.mjs
💤 Files with no reviewable changes (1)
  • .fas/TASKS.md
✅ Files skipped from review due to trivial changes (1)
  • .fas/tasks/build-a-headless-anthropic-tool-use-dogfood-example-for-igni.md
🚧 Files skipped from review as they are similar to previous changes (2)
  • scripts/tests/test-examples.test.mjs
  • scripts/test-examples.mjs

Walkthrough

The PR splits the root test command into package, script, example, and full lanes, adds a script that discovers and runs example runtime tests, updates CI and FAS config, and adds new FAS tracker entries and task documents for igniteTools work.

Changes

Example Runtime Test Lane Split

Layer / File(s) Summary
Example test discovery and execution
scripts/test-examples.mjs, scripts/__tests__/test-examples.test.mjs
The new CLI discovers runtime tests under examples/, derives example roots, checks configs, installs missing dependencies when needed, runs Vitest per root, and exits with failure on any error; tests cover --list output and orphaned runtime tests.
Test lane wiring and CI
package.json, .fas-config.json, .github/workflows/ci.yml
The root test scripts now split into test:packages, test:scripts, test:examples, and test:full; FAS points at test:full; CI adds test:scripts and test:examples steps after coverage.

FAS Task Planning Documents

Layer / File(s) Summary
TASKS.md tracker entries
.fas/TASKS.md
The tracker gains new igniteTools entries for observe(), the Anthropic ToolDialect adapter, the dogfood example, showcase phases A/B/C, the host-type split, PR2 follow-ups, and the example test-lane fix.
Task specification documents
.fas/tasks/*
New task packets describe the example test-lane fix and the igniteTools observe, ToolDialect, dogfood, showcase, host-type split, and PR2 follow-up work, including acceptance criteria and verification steps.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~15 minutes

Possibly related PRs

  • 0xjcf/ignite-element#69: Shares the same createRuntimeHost DOM-free headless runtime direction referenced by the new Phase A task packet.
  • 0xjcf/ignite-element#71: Also changes the root test scripting around example runtime coverage, which this PR now restructures into dedicated lanes.

Poem

🐇 I hopped through lanes both split and neat,
test:full now keeps the rhythm sweet.
Example roots dance in orderly lines,
CI and FAS now share the signs.
Hop-hop hooray, the tasks are queued!

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main change: fixing the example runtime test lanes.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fas/example-runtime-test-lanes

Comment @coderabbitai help to get the list of available commands.

@0xjcf
0xjcf marked this pull request as ready for review June 29, 2026 16:09

0xjcf commented Jun 29, 2026

Copy link
Copy Markdown
Owner Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jun 29, 2026

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@codecov

codecov Bot commented Jun 29, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (beta@4248585). Learn more about missing BASE report.

Additional details and impacted files

Impacted file tree graph

@@           Coverage Diff           @@
##             beta      #73   +/-   ##
=======================================
  Coverage        ?   89.96%           
=======================================
  Files           ?       32           
  Lines           ?     1515           
  Branches        ?      435           
=======================================
  Hits            ?     1363           
  Misses          ?      142           
  Partials        ?       10           
Flag Coverage Δ
unittests 89.96% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.fas/TASKS.md:
- Around line 1634-1643: The task tracker contains duplicate Phase A/Phase B
rows for the same briefs/slugs, which creates conflicting state in FAS. Update
the existing task entries in the TASKS.md tracker in place or remove the
duplicate rows for the igniteTools showcase Phase A and Phase B items, using the
task title/brief/slugs to identify the records that already exist above.

In @.fas/tasks/build-a-headless-anthropic-tool-use-dogfood-example-for-igni.md:
- Line 1: The task title is truncated and ends at “pluggable-”, so complete the
heading text in the markdown task file so it reads as a full searchable title.
Update the existing task title only, keeping the same intent and using the same
task heading so it is no longer cut off.

In `@scripts/test-examples.mjs`:
- Around line 104-107: The example discovery flow in test-examples.mjs is
silently filtering out test files whose parent chain does not resolve to a valid
example root, which can hide orphaned runtime tests. Update the
findTestFiles/findExampleRoot pipeline so these orphaned .test.* files are
detected before building exampleRoots and cause a hard failure with a clear
error, instead of being dropped by the Boolean filter; keep the validation close
to the exampleRoots construction and the findExampleRoot mapping.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: ab294c1f-9304-4123-851c-97a30954d968

📥 Commits

Reviewing files that changed from the base of the PR and between 4248585 and f128eff.

📒 Files selected for processing (15)
  • .fas-config.json
  • .fas/TASKS.md
  • .fas/tasks/add-an-observe-observation-channel-to-the-neutral-ignitetool.md
  • .fas/tasks/build-a-headless-anthropic-tool-use-dogfood-example-for-igni.md
  • .fas/tasks/fix-example-runtime-test-lanes-before-next-ignitetools-task.md
  • .fas/tasks/ignitetools-pr2-coderabbit-follow-ups-bind-runtime-execute-s.md
  • .fas/tasks/ignitetools-showcase-phase-a-make-the-headless-agent-runtime.md
  • .fas/tasks/ignitetools-showcase-phase-b-headless-smart-home-agent-examp.md
  • .fas/tasks/ignitetools-showcase-phase-c-terminal-to-browser-bridge-head.md
  • .fas/tasks/implement-ignitetools-anthropic-tooldialect-adapter-at-ignit.md
  • .fas/tasks/split-the-agent-runtime-host-type-eventtarget-from-the-rende.md
  • .github/workflows/ci.yml
  • package.json
  • scripts/__tests__/test-examples.test.mjs
  • scripts/test-examples.mjs

Comment thread .fas/TASKS.md Outdated
Comment thread .fas/tasks/build-a-headless-anthropic-tool-use-dogfood-example-for-igni.md Outdated
Comment thread scripts/test-examples.mjs Outdated

0xjcf commented Jun 29, 2026

Copy link
Copy Markdown
Owner Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jun 29, 2026

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

0xjcf commented Jun 29, 2026

Copy link
Copy Markdown
Owner Author

Read the remaining CodeRabbit pre-merge docstring coverage warning. Leaving it as non-blocking for this PR: this repo does not have a docstring coverage gate, the changed runner follows the existing script style, and local/FAS verification plus CI pass. The three actionable inline findings were fixed in d3b52a9 and resolved.

@0xjcf
0xjcf merged commit be27a9c into beta Jun 29, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant