feat: model and host adapters (release evaluation PR 3) - #40
Closed
noodlemind wants to merge 1 commit into
Closed
Conversation
Controlled Kimi/OpenRouter host with fail-closed credentials, local Gemma/Ollama host with Docker endpoint rewrite, Codex and Claude subscription A/B contracts with transcript preservation and null-not- estimated telemetry normalization, and Copilot/Grok compatibility smoke checklists with fail-closed evaluation.
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
Owner
Author
|
Consolidated into #38 per review direction — all four workstreams ship as one PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PR 3 of the release-evaluation plan: the model and host adapters that the release orchestrator (PR 4) schedules.
openrouter-kimi.mjs— the controlled API experiment on the pinned Kimi K2.7 Code profile. Credential validation fails closed: noOPENROUTER_API_KEY, no driver, no spend. Gates the release after calibration.ollama-gemma.mjs— the local capability floor: free, credential-less, informational gate, with the Docker-container endpoint rewrite (host.docker.internal) from the plan.codex-subscription.mjs/claude-subscription.mjs— the manual frontier A/B contracts (scheduled rotation). Each defines operator run instructions (fresh sandboxes, record the exact resolved model, never mix numerically with the OpenRouter result), transcript preservation to disk, and host-report normalization where every unavailable or malformed telemetry field is recorded asnull— never silently estimated.copilot-smoke.mjs/grok-smoke.mjs— the host-compatibility checklist (install/hydration, agent discovery, skill activation, hook behavior, completion+verifier). Evaluation fails closed: unreported checklist items count as failures.Test coverage
9 new deterministic tests (
eval-hosts.test.mjs): pinned-profile wiring, fail-closed credential validation, Docker endpoint rewrite, the all-null subscription telemetry template, normalization of malformed host reports, transcript preservation, and fail-closed smoke evaluation.Stack
Based on #39 (PR 2: Terminal-Bench adapter), which is based on #38 (PR 1). PR 4 adds release orchestration and reporting.