Skip to content

test: add deterministic run-with-first-tree eval floor - #6

Merged
Gandy2025 merged 24 commits into
mainfrom
feat/run-with-first-tree-live-gate
Aug 6, 2026
Merged

test: add deterministic run-with-first-tree eval floor#6
Gandy2025 merged 24 commits into
mainfrom
feat/run-with-first-tree-live-gate

Conversation

@Gandy2025

@Gandy2025 Gandy2025 commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add a deterministic five-case evaluation Floor for run-with-first-tree
  • validate the immutable case matrix, strict fixture schema, and independently specified graph facts
  • materialize every fixture and exercise the real local builder-to-validator CLI seam
  • run the complete Floor in CI on Python 3.10 and 3.12

Scope

The Floor covers the graph inputs behind five important future behavior scenarios:

  • unmanaged dry run
  • invalid dependency cycle
  • untrusted ticket prose
  • ambiguous create recovery
  • forged non-primary outcome

For this PR, those names preserve the intended future evaluation matrix. The deterministic oracle is limited to case coverage, fixture shape, and builder-to-validator graph facts.

Boundaries

  • no model or provider runner
  • no First Tree CLI, Cloud, Chat, or tracker calls
  • no credentials, network sandbox, recording shim, transcript parser, model grader, run manifest, or retained provider artifacts
  • no claim about live agent behavior, dispatch safety, final responses, public release, or the earlier staging Pilot
  • the product Skill payload under skills/ is byte-identical to the base

A model-backed Gate is intentionally deferred until it can use a separately reviewed evidence architecture.

Validation

  • exact head: ac8db30c3f1d186e5d04ee24fa852e758a135bfa
  • Python 3.10: structure validator, 78 unit tests, deterministic five-case Floor, compileall
  • Python 3.12: structure validator, 78 unit tests, deterministic five-case Floor, compileall
  • uvx ruff check .
  • uvx ruff format --check .
  • uvx --from skills-ref agentskills validate skills/run-with-first-tree
  • npx --yes skills@1.5.21 add . --list
  • git diff --check

@Gandy2025 Gandy2025 changed the title test: add run-with-first-tree live gate test: add deterministic run-with-first-tree eval floor Aug 6, 2026
@Gandy2025
Gandy2025 marked this pull request as ready for review August 6, 2026 04:52
@Gandy2025
Gandy2025 merged commit 4c10cc3 into main Aug 6, 2026
3 checks passed
@Gandy2025
Gandy2025 deleted the feat/run-with-first-tree-live-gate branch August 7, 2026 09:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant