Found by the Agents on Guren benchmark (20 tasks × 3 models × bare/shipped harness × 3 trials).
The API-signature digest emitted by guren context (and injected by the agent:init SessionStart hook) does not mention:
- the health-check subsystem (
createHealthManager, DatabaseCheck, HealthManager.middleware(), critical), or
- the redirect-safety helpers (
isSafeRedirectUrl, sanitizeOAuthRedirect, createRedirectSafetyMiddleware).
Measured consequence: in the benchmark's open-redirect-login task, 0 of 17 passing runs in either condition used isSafeRedirectUrl — every solution hand-rolled the check; in health-db-probe, no run was API-only (the ones that found createHealthManager still hand-wrote SELECT 1). APIs the digest names (validateBody, authorize, redirect, paginate, resource fields) were used 9/9 in both conditions. The digest is what decides whether an agent reaches for the framework API; what it omits stays undiscovered even with 120 turns.
Suggested fix: add both subsystems to the digest in packages/cli/src/context.ts (short signature lines + the "critical: true" gotcha for the health manager), and re-run those two tasks.
Found by the Agents on Guren benchmark (20 tasks × 3 models × bare/shipped harness × 3 trials).
The API-signature digest emitted by
guren context(and injected by theagent:initSessionStart hook) does not mention:createHealthManager,DatabaseCheck,HealthManager.middleware(),critical), orisSafeRedirectUrl,sanitizeOAuthRedirect,createRedirectSafetyMiddleware).Measured consequence: in the benchmark's
open-redirect-logintask, 0 of 17 passing runs in either condition usedisSafeRedirectUrl— every solution hand-rolled the check; inhealth-db-probe, no run was API-only (the ones that foundcreateHealthManagerstill hand-wroteSELECT 1). APIs the digest names (validateBody,authorize,redirect,paginate, resource fields) were used 9/9 in both conditions. The digest is what decides whether an agent reaches for the framework API; what it omits stays undiscovered even with 120 turns.Suggested fix: add both subsystems to the digest in
packages/cli/src/context.ts(short signature lines + the "critical: true" gotcha for the health manager), and re-run those two tasks.