ci: run daily Luna agent evals - #331
Merged
Merged
Conversation
Run workload executions through a bounded in-process pool, persist the execution bound in run and suite artifacts, and preserve legacy suite readers and comparisons.
Render schema-validated suite telemetry as a compact absolute report and preserve failure diagnostics before returning a failing status.
Run bounded Luna discovery and intent suites in clean GitHub Actions jobs, retain raw evidence, and render an absolute CI summary.
Document bounded workload pools, manifest ordering, and workflow-run evidence links after the CI concurrency implementation.
Write a non-secret scenario marker before checkout so artifact uploads remain non-empty when setup or authentication fails.
Prevent child execution for suites with no selected workloads, fail zero-execution CI evidence, and leave enough timeout margin for the six intent waves.
Keep operational documentation consistent with the 40-minute scenario job timeout while preserving the separate five-minute workload timeout.
Align Phase 3 failure enumerations and suite preflight documentation with zero-workload validation.
jlitola
added
the
agent-eval
Pass the GitHits token through Codex MCP env_vars by name so stdio MCP children inherit authentication without persisting secret values.
Keep Codex MCP env_vars in the server table before its literal environment subtable so generated TOML remains valid.
jlitola
added
agent-eval
Document the failed and corrected label-triggered runs, measured CI timings and costs, and the remaining post-merge scheduled/manual validation.
Keep live label evidence distinct from the trusted manual and scheduled default-branch checks that remain pending after merge.
jlitola
added
agent-eval
jlitola
marked this pull request as ready for review
August 31, 2026 10:34
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
GITHITS_API_TOKENto Codex MCP by variable name without persisting its valueValidation
Live validation
dc63675: discovery 2/2 and intent 21/21, MCP-only tool use, no warnings16fd964and provided the recorded timing, token, and cost evidenceScheduled/manual default-branch validation remains pending merge because the workflow does not exist on
mainyet.