Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
62 changes: 62 additions & 0 deletions .proofs/ARN-50-db-latency-2026-06-18.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
# Proof Report: ARN-50 — Foresight DB Hot-Path Linkage Fix

## Date

2026-06-18

## Branch / Commit

Branch: `codex/arn50-db-latency`

Implementation commit: `ba625e44`

## What Was Done

Re-verified the canonical `foresight` deployment filter in Datadog and implemented the first focused fix for the verified denied raw `PATCH /tdata/Paths` class:

- Added governed `Path.AssignRepairer`, `Path.AssignAdversary`, and `Path.AppendChallengeFlag` actions.
- Permitted those actions for system principals only.
- Updated `spawn_repairers`, `spawn_adversaries`, and `animate_dwellers` to dispatch bound actions instead of raw Path PATCH.
- Added contract and Cedar tests for the new action surface.

## Verification Flow

## Verification Results
| Step | Expected | Actual | Status |
|------|----------|--------|--------|
| Datadog trace decode | Identify canonical Foresight/Supabase filters | `@version:sha-foresight-dd-638ff9b1` and `@peer.service:foresight-supabase` matched; plain `version:` returned 0 buckets | Pass |
| Foresight-only 10m APM sample | Quantify request/query/internal span amplification | 81 HTTP request spans, 1,218 Postgres query spans, 28,305 internal spans | Pass |
| DB span wait/busy split | Confirm DB path is mostly wait, not compute | Hot query averages were ~136-487 ms wall with ~0.11-2.67 ms busy; idle time dominated | Pass |
| Corridor spec/Cedar tests | New actions exist and are system-only | `cargo test -p temperpaw --test corridor_engine_contract --test corridor_cedar_matrix` passed: 25 tests | Pass |
| Edited WASM unit tests | Edited modules still compile and pass unit contracts | `spawn_repairers` 12/12, `spawn_adversaries` 7/7, `animate_dwellers` 6/6 passed | Pass |
| Foresight WASM bundle | All app WASM modules build for `wasm32-unknown-unknown` | `bash os-apps/paw-foresight/wasm/build.sh` completed; all 13 modules built | Pass |

## What Worked
- `@version` and `@peer.service` are the working Datadog filters for this lane.
- The denied Path PATCH class maps directly to app-state updates and can be replaced with entity actions.

## What Didn't Work
- Plain `version:` APM filtering returned zero buckets even though spans carry `version` in custom attributes.
- DBM sample lookup did not return activity rows for the inspected window.

## Limitations

No live deployment was changed in this thread. The fix is verified locally by tests and WASM build, not by a fresh production run.

## What Still Doesn't Work

The larger DB bottleneck is still snapshot/projection/catalog/index write amplification. This fix removes a confirmed denial-recording waste class, but it is not the primary current latency lever.

## Artifacts

- Datadog trace: `2e06e546a5184dcc284c496f78e9ca86`
- Working branch: `codex/arn50-db-latency`

## Architecture Diagram
```text
spawn_repairers ── Path.AssignRepairer ──▶ Path(Solving)
spawn_adversaries ─ Path.AssignAdversary ▶ Path(Repaired)
animate_dwellers ─ Path.AppendChallengeFlag ▶ Path(Scored/Canonical/Tail)

No raw PATCH /tdata/Paths for these hot-linkage updates.
```
26 changes: 26 additions & 0 deletions crates/temperpaw/tests/corridor_cedar_matrix.rs
Original file line number Diff line number Diff line change
Expand Up @@ -143,6 +143,32 @@ fn only_the_assigned_repairer_and_adversary_self_report() {
);
}

#[test]
fn path_linkage_actions_are_system_only() {
let engine = engine();
let a = attrs(&[
("id", serde_json::json!("p-1")),
("RepairerAgentId", serde_json::json!("rep-1")),
("AdversaryAgentId", serde_json::json!("adv-1")),
]);

let system = ctx("spawn-wasm", "system");
for action in ["AssignRepairer", "AssignAdversary", "AppendChallengeFlag"] {
assert!(
engine.authorize(&system, action, "Path", &a).is_allowed(),
"system WASM must dispatch Path.{action}"
);
}

let session = ctx("rep-1", "agent");
for action in ["AssignRepairer", "AssignAdversary", "AppendChallengeFlag"] {
assert!(
!engine.authorize(&session, action, "Path", &a).is_allowed(),
"sessions must not dispatch Path.{action}"
);
}
}

#[test]
fn claim_amendment_is_relay_writable_but_verdicts_are_system_only() {
// ADR-004: the repairer (relayed as service:wasm-runtime) may amend a
Expand Down
36 changes: 36 additions & 0 deletions crates/temperpaw/tests/corridor_engine_contract.rs
Original file line number Diff line number Diff line change
Expand Up @@ -384,6 +384,42 @@ fn path_scoring_is_deterministic_wasm_territory() {
assert!(action_params(score).contains("repair_cost"));
}

#[test]
fn path_hot_linkage_updates_are_governed_actions_not_raw_patch() {
let path = spec_path("path.ioa.toml");
let spec = parse_spec(&path);

let repairer = action(&spec, "AssignRepairer", &path);
assert_eq!(
action_from(repairer),
["Solving".to_string()].into_iter().collect(),
"repairer assignment happens while a route is solving"
);
assert!(action_params(repairer).contains("repairer_agent_id"));

let adversary = action(&spec, "AssignAdversary", &path);
assert_eq!(
action_from(adversary),
["Repaired".to_string()].into_iter().collect(),
"adversary assignment happens before challenge"
);
assert!(action_params(adversary).contains("adversary_agent_id"));

let append = action(&spec, "AppendChallengeFlag", &path);
assert_eq!(
action_from(append),
[
"Scored".to_string(),
"Canonical".to_string(),
"Tail".to_string()
]
.into_iter()
.collect(),
"dweller stress tests append flags only after route costing/classification"
);
assert!(action_params(append).contains("challenge_flags"));
}

#[test]
fn endpoint_decomposition_bridges_bundles_to_claims() {
// ADR-004: SubmitForRepair no longer spawns repairers directly — it
Expand Down
27 changes: 27 additions & 0 deletions os-apps/paw-foresight/adrs/009-governed-path-hot-linkage.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# ADR-009: Governed Path Hot-Linkage Actions

## Status

Accepted

## Context

Datadog traces for the canonical `foresight` deployment on Supabase showed a repeated denied raw `PATCH /tdata/Paths('{id}')` class on the corridor hot path. The request was cheap in CPU time but expensive in wall time because each denial still recorded governance decisions and denial patterns while the Supabase pool was already contended.

The affected updates were not business-state shortcuts. They linked newly spawned repairer/adversary agents to a `Path` and appended dweller contradiction flags after costing. Those changes are part of the app state machine and should be visible as governed transitions.

## Decision

Add narrow `Path` actions for these hot-linkage updates:

- `AssignRepairer(repairer_agent_id)` while `Path` is `Solving`.
- `AssignAdversary(adversary_agent_id)` while `Path` is `Repaired`.
- `AppendChallengeFlag(challenge_flags)` while `Path` is `Scored`, `Canonical`, or `Tail`.

Only system principals may dispatch these actions. Foresight WASM modules use bound `TemperPaw.*` action dispatches instead of raw OData `PATCH` for these fields.

## Consequences

The denied-PATCH class is removed from this corridor path, and assignment failures now fail before spawning sessions instead of silently loosening the assigned-agent self-report guard.

This does not solve the larger current write-amplification bottleneck from snapshot, catalog, projection, and index writes. That remains the higher-leverage DB-path work before any inference concurrency widening.
3 changes: 2 additions & 1 deletion os-apps/paw-foresight/policies/foresight.cedar
Original file line number Diff line number Diff line change
Expand Up @@ -247,7 +247,8 @@ permit(
action in [Action::"Score", Action::"ClassifyCanonical", Action::"ClassifyTail",
Action::"Reject", Action::"Rescore", Action::"RequestChallenge",
Action::"Prune", Action::"RevisionRequested", Action::"ResumeRepair",
Action::"ResumeCosting"],
Action::"ResumeCosting", Action::"AssignRepairer",
Action::"AssignAdversary", Action::"AppendChallengeFlag"],
resource is Path
) when {
principal.agent_type == "system"
Expand Down
21 changes: 21 additions & 0 deletions os-apps/paw-foresight/specs/path.ioa.toml
Original file line number Diff line number Diff line change
Expand Up @@ -198,6 +198,13 @@ on_failure = "Fail"
[action.triggers.config]
temper_api_url = "{secret:temper_api_url}"

[[action]]
name = "AssignRepairer"
kind = "input"
from = ["Solving"]
params = ["repairer_agent_id"]
hint = "Bind the freshly spawned repairer agent to this route. System principals only; replaces raw PATCH on the hot DB path."


[[action]]
name = "RequestChallenge"
Expand All @@ -216,6 +223,13 @@ on_failure = "Fail"
[action.triggers.config]
temper_api_url = "{secret:temper_api_url}"

[[action]]
name = "AssignAdversary"
kind = "input"
from = ["Repaired"]
params = ["adversary_agent_id"]
hint = "Bind the freshly spawned adversary agent to this route. System principals only; replaces raw PATCH on the hot DB path."


[[action]]
name = "Prune"
Expand Down Expand Up @@ -324,6 +338,13 @@ to = "Scored"
params = ["repair_cost"]
hint = "World update re-derived this path's cost (a resolved EventNode moved the graph). System principals only."

[[action]]
name = "AppendChallengeFlag"
kind = "input"
from = ["Scored", "Canonical", "Tail"]
params = ["challenge_flags"]
hint = "Dweller stress-test append of the route's challenge flags using the full rewritten flag JSON. System principals only; replaces raw PATCH."

[[action]]
name = "Fail"
kind = "input"
Expand Down
22 changes: 9 additions & 13 deletions os-apps/paw-foresight/wasm/animate_dwellers/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -816,24 +816,20 @@ fn phase_contradiction(ctx: &Context, fields: &Value) -> Result<(), String> {
"severity": "low",
"note": format!("dweller contradiction ({}): {note}", ctx.entity_id),
}));
let patch = json!({
let append = json!({
"challenge_flags": serde_json::to_string(&flags).unwrap_or_default()
});
let r = ctx.http_call(
"PATCH",
&format!("{api}/tdata/Paths('{path_id}')"),
let r = dispatch(
ctx,
&api,
&headers,
&patch.to_string(),
"Paths",
&path_id,
"AppendChallengeFlag",
&append,
);
match r {
Ok(resp) if resp.status < 400 => {}
Ok(resp) => ctx.log(
"warn",
&format!(
"animate_dwellers: challenge-flag append on {path_id} failed (HTTP {})",
resp.status
),
),
Ok(()) => {}
Err(e) => ctx.log(
"warn",
&format!("animate_dwellers: challenge-flag append on {path_id} failed: {e}"),
Expand Down
61 changes: 36 additions & 25 deletions os-apps/paw-foresight/wasm/spawn_adversaries/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -185,6 +185,32 @@ fn create_agent(
.ok_or_else(|| "Agent create returned no entity_id".to_string())
}

fn dispatch_action(
ctx: &Context,
api: &str,
headers: &[(String, String)],
set: &str,
id: &str,
action: &str,
body: &Value,
) -> Result<(), String> {
let resp = ctx.http_call(
"POST",
&format!("{api}/tdata/{set}('{id}')/TemperPaw.{action}"),
headers,
&body.to_string(),
)?;
if (200..300).contains(&resp.status) {
Ok(())
} else {
Err(format!(
"{set}('{id}').{action} failed (HTTP {}): {}",
resp.status,
&resp.body[..resp.body.len().min(200)]
))
}
}

#[allow(clippy::too_many_arguments)]
fn start_session(
ctx: &Context,
Expand Down Expand Up @@ -351,39 +377,24 @@ pub extern "C" fn run(_ctx_ptr: i32, _ctx_len: i32) -> i32 {
}
};

// Create the adversary agent and bind it to the path. A PATCH failure
// only loosens Cedar's assigned-adversary check: warn and proceed.
// Create the adversary agent and bind it to the path before session
// spawn so Path.ChallengeComplete remains assigned-agent guarded.
let adversary_agent_id = create_agent(
&ctx,
&api,
&headers,
&format!("Adversary-{path_id}"),
"adversary",
)?;
let patch_body = json!({ "adversary_agent_id": adversary_agent_id });
match ctx.http_call(
"PATCH",
&format!("{api}/tdata/Paths('{path_id}')"),
dispatch_action(
&ctx,
&api,
&headers,
&patch_body.to_string(),
) {
Ok(r) if r.status < 400 => {}
Ok(r) => ctx.log(
"warn",
&format!(
"spawn_adversaries: PATCH Paths('{path_id}') adversary_agent_id failed \
(HTTP {}); Cedar's assigned-adversary check won't bind",
r.status
),
),
Err(e) => ctx.log(
"warn",
&format!(
"spawn_adversaries: PATCH Paths('{path_id}') adversary_agent_id failed \
({e}); Cedar's assigned-adversary check won't bind"
),
),
}
"Paths",
&path_id,
"AssignAdversary",
&json!({ "adversary_agent_id": adversary_agent_id }),
)?;

// Spawn the adversary session.
let adversary_msg = adversary_prompt(
Expand Down
Loading