Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 9 additions & 1 deletion crates/paw-codex-worker/src/cli.rs
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,15 @@ fn parse_worker_command(args: impl IntoIterator<Item = String>) -> WorkerCommand
return WorkerCommand::LaunchdPlist;
}
"run" | "--run" => return WorkerCommand::Run,
"directed-evolution-demo" | "--directed-evolution-demo" => {
return WorkerCommand::DirectedEvolutionDemo;
}
"directed-evolution-run" | "--directed-evolution-run" => {
return WorkerCommand::DirectedEvolutionRun;
}
"directed-evolution-mutate" | "--directed-evolution-mutate" => {
return WorkerCommand::DirectedEvolutionMutate;
}
_ => {}
}
}
Expand Down Expand Up @@ -48,4 +57,3 @@ fn value_as_bool(value: &Value) -> Option<bool> {
.map(|value| value.eq_ignore_ascii_case("true"))
})
}

552 changes: 552 additions & 0 deletions crates/paw-codex-worker/src/directed_evolution.rs

Large diffs are not rendered by default.

64 changes: 64 additions & 0 deletions crates/paw-codex-worker/src/directed_evolution_tests.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
#[test]
fn live_evolution_requires_immutable_genesis_refs() {
assert_eq!(
evolution_ref("MISSING_REF", "demo/agent-answers@seed", false).expect("smoke ref"),
"demo/agent-answers@seed"
);
let error = evolution_ref("MISSING_REF", "demo/agent-answers@seed", true)
.expect_err("live evolution must reject a label ref");
assert!(format!("{error:#}").contains("immutable Genesis ref"));
}

#[test]
fn evolution_candidate_changes_are_limited_to_native_bundle_files() {
assert!(evolution_candidate_path_allowed("specs/answer.ioa.toml"));
assert!(evolution_candidate_path_allowed("wasm/validator/src/lib.rs"));
assert!(evolution_candidate_path_allowed("adrs/0002-evidence.md"));
assert!(!evolution_candidate_path_allowed("crates/random-helper/src/lib.rs"));
assert!(!evolution_candidate_path_allowed("../evaluator/specs/trial.ioa.toml"));
}

#[test]
fn generic_evolution_plan_allows_subject_defined_metrics_and_traffic() {
let plan: EvolutionCampaignPlan = serde_json::from_str(
r#"{"campaign_id":"campaign-support","name":"Support Inbox","director_brief":"Improve resolution.","target_app_ref":"owner/support@1111111111111111111111111111111111111111","evaluator_app_ref":"owner/support-eval@2222222222222222222222222222222222222222","brain_provider":"codex","automation_mode":"automatic_release","traffic_sources":[{"id":"ticket-stream","name":"tickets","kind":"real","description":"incoming support tickets"}],"selection_design":{"id":"support-selection","version_label":"v1","evaluator_namespace":"Acme.SupportEvaluation","trial_suite":{"id":"support-suite","name":"Triage","description":"Resolve urgent cases.","scenario_manifest_json":[{"id":"urgent-ticket"}],"hidden_fixture_locator":"temper://fixture","authored_by":"codex"},"fitness_model_json":{"comparison":"preference","signals":["resolution_quality"]},"constraint_definitions_json":[],"traffic_sources_json":["tickets"],"rationale":"Prefer resolved cases.","proposed_by":"codex","approved_by":"human","metrics":[{"id":"resolution-metric","key":"resolution_quality","description":"quality","instrument_kind":"native","instrument_locator":"temper://quality","interpretation":"higher is better","hard_constraint":false}]},"generations":[{"ordinal":"1","parent_release_ref":"owner/support@1111111111111111111111111111111111111111","selected_app_ref":"owner/support@3333333333333333333333333333333333333333"}],"release_control":{"pause_reason":"inspect","rollback_current_ref":"owner/support@1111111111111111111111111111111111111111","rollback_previous_ref":"owner/support@3333333333333333333333333333333333333333","rollback_reason":"rollback"}}"#,
)
.expect("generic campaign plan should parse");
assert_eq!(plan.selection_design.metrics[0].key, "resolution_quality");
assert_eq!(plan.selection_design.evaluator_namespace, "Acme.SupportEvaluation");
assert_eq!(plan.traffic_sources[0].name, "tickets");
}

#[tokio::test]
async fn live_evolution_binds_releases_to_executed_validator_evidence() {
let _guard = ENV_LOCK.lock().await;
let path = unique_temp_dir().join("validator-evidence.json");
fs::create_dir_all(path.parent().expect("manifest parent")).expect("manifest parent");
fs::write(
&path,
r#"{"evaluator_ref":"owner/evaluator@aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa","records":[{"generation":"1","candidate_ref":"owner/app@1111111111111111111111111111111111111111","status":"Passed","evidence_locator":"temper://trial/g1/validator","result_summary":"generation one passed","measurements":[{"suffix":"quality","traffic_source_id":"simulated","metric_key":"quality","metric_value":"0.8","source_kind":"simulated","evidence_locator":"temper://trial/g1/validator"}]},{"generation":"2","candidate_ref":"owner/app@2222222222222222222222222222222222222222","status":"Passed","evidence_locator":"temper://trial/g2/validator","result_summary":"generation two passed","measurements":[{"suffix":"retention","traffic_source_id":"real","metric_key":"workflow_completion","metric_value":"0.92","source_kind":"real","evidence_locator":"temper://trial/g2/validator"}]}]}"#,
)
.expect("validator evidence fixture");
let _evidence = EnvOverride::set(
"EVOLUTION_VALIDATOR_EVIDENCE_PATH",
path.as_os_str().to_os_string(),
);
let records = load_evolution_validation_evidence(
"owner/evaluator@aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
&[
("1", "owner/app@1111111111111111111111111111111111111111"),
("2", "owner/app@2222222222222222222222222222222222222222"),
],
true,
)
.expect("matching executed evidence");
assert_eq!(records[1].measurements[0].metric_key, "workflow_completion");

let error = load_evolution_validation_evidence(
"owner/evaluator@aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
&[("2", "owner/app@3333333333333333333333333333333333333333")],
true,
)
.expect_err("unexecuted candidate must not release");
assert!(format!("{error:#}").contains("validator evidence missing generation 2"));
}
10 changes: 10 additions & 0 deletions crates/paw-codex-worker/src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,15 @@ async fn main() -> Result<()> {
);
return Ok(());
}
if command == WorkerCommand::DirectedEvolutionDemo {
return run_directed_evolution_demo(&client, &config).await;
}
if command == WorkerCommand::DirectedEvolutionRun {
return run_directed_evolution_run(&client, &config).await;
}
if command == WorkerCommand::DirectedEvolutionMutate {
return run_directed_evolution_mutation(&config).await;
}

info!(
worker_id = %config.worker_id,
Expand Down Expand Up @@ -103,5 +112,6 @@ include!("codex_plan.rs");
include!("code_evaluation.rs");
include!("execution.rs");
include!("http_headers.rs");
include!("directed_evolution.rs");
include!("tests.rs");
include!("daily_brief_tests.rs");
13 changes: 13 additions & 0 deletions crates/paw-codex-worker/src/tests.rs
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ mod tests {
include!("codex_plan_tests.rs");
include!("pull_request_tests.rs");
include!("worker_http_tests.rs");
include!("directed_evolution_tests.rs");

static ENV_LOCK: Mutex<()> = Mutex::const_new(());

Expand Down Expand Up @@ -226,6 +227,18 @@ mod tests {
parse_worker_command(vec!["run".to_string()]),
WorkerCommand::Run
);
assert_eq!(
parse_worker_command(vec!["directed-evolution-demo".to_string()]),
WorkerCommand::DirectedEvolutionDemo
);
assert_eq!(
parse_worker_command(vec!["directed-evolution-run".to_string()]),
WorkerCommand::DirectedEvolutionRun
);
assert_eq!(
parse_worker_command(vec!["directed-evolution-mutate".to_string()]),
WorkerCommand::DirectedEvolutionMutate
);
}

#[test]
Expand Down
10 changes: 10 additions & 0 deletions crates/paw-codex-worker/src/worker_types.rs
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,9 @@ enum WorkerCommand {
Run,
Doctor,
LaunchdPlist,
DirectedEvolutionDemo,
DirectedEvolutionRun,
DirectedEvolutionMutate,
}

#[derive(Clone, Copy, Debug, PartialEq, Eq)]
Expand Down Expand Up @@ -133,6 +136,13 @@ impl Config {
self.temper_url, entity_set, id, action
)
}

fn namespaced_action_url(&self, namespace: &str, entity_set: &str, id: &str, action: &str) -> String {
format!(
"{}/tdata/{}('{}')/{}.{}",
self.temper_url, entity_set, id, namespace, action
)
}
}

fn load_worker_env_file() -> Result<()> {
Expand Down
49 changes: 49 additions & 0 deletions docs/adrs/0052-codex-directed-evolution-brain-provider.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
# ADR 0052: Codex as the Directed-Evolution V1 Brain Provider

## Status

Accepted.

## Decision

V1 runs the directed-evolution brain through `paw-codex-worker`. The
`directed-evolution-run` mode consumes an `EVOLUTION_CAMPAIGN_PLAN_PATH`
manifest so arbitrary Temper-native subjects can supply their own traffic,
trial suite, metrics, capability decisions, generations and release controls.
The plan also declares its evaluator namespace and entity-set names, so the
runner does not depend on the Agent Answers evaluator namespace.
The `directed-evolution-demo` mode remains an Agent Answers convenience entry
point and can call Codex for a selection-design rationale when
`PAW_EVOLUTION_USE_CODEX=1`.
Deterministic smoke mode exercises the protocol without an external model call.
The `directed-evolution-mutate` mode asks Codex to edit a candidate workspace,
then rejects any change outside the Temper-native subject app directories
before Genesis publishes or installs that candidate. Its frozen evaluator
compatibility contract is campaign input, rather than a built-in dependency on
the Agent Answers interaction model.
Live Codex mode requires the seed and both selected candidate versions to be
immutable Genesis commit refs (`owner/app@hash`); it does not accept illustrative
candidate labels as releases.

The worker records Codex as a provider and communicates only through native
campaign actions. It does not embed a fixed fitness vector or mutate the
active evaluator while candidate trials are running. A future TemperPaw-native
brain can issue the same actions and replace Codex without changing campaign
state or Evolution Studio.

## Evidence And Release Control

The proof mode freezes evaluator-owned `TrialSuite` and `MetricDefinition`
records. A live run requires `EVOLUTION_VALIDATOR_EVIDENCE_PATH`, produced by
executing the frozen scenario against the exact pinned candidate refs; a
mismatched or absent record prevents release. The worker records a native
`ValidatorRun` for each validated selected candidate and attaches
simulated, real-traffic, and Datadog evidence locators, performs two automatic
local releases, then pauses and rolls back. New local
Datadog ingestion requires an execution-time `DD_API_KEY`; absent that key the
Datadog locator remains explicitly pending instead of claiming ingestion.

The paired Genesis lineage smoke publishes and installs two real Temper-native
subject versions before these refs are handed to this runner. This separation
keeps candidate bytes and installability in Genesis while campaign decisions and
human direction remain native directed-evolution records.