Roadmap priority: B7
Summary
Automatically log every prompt, response, model, and token count per BehaviorSpace run to CSV, so LLM behaviour can be analysed alongside model outputs.
Why it matters
Every demo that needed this built it by hand — demos/provider-sensitivity/ has its own CSV writing and a Python analysis script, and crisis-triage does the same. That per-demo scaffolding should be a feature.
Without it, a sweep tells you what happened in the model but not what the LLM said, so there is no way to correlate outcomes with prompts or drift.
What needs to be done
- Detect BehaviorSpace context and the current run number from the workspace.
- Auto-write a per-run CSV: run number, tick, agent id, prompt, response, model, provider, token counts, latency, error (if any).
- Config key to enable and to set the output path; off by default.
- Handle concurrent BehaviorSpace runs writing to the same target without interleaving corruption.
- Escape newlines and quotes properly — prompts and responses are multi-line free text, which naive CSV writing will break.
Open questions
Related
Roadmap priority: B7
Summary
Automatically log every prompt, response, model, and token count per BehaviorSpace run to CSV, so LLM behaviour can be analysed alongside model outputs.
Why it matters
Every demo that needed this built it by hand —
demos/provider-sensitivity/has its own CSV writing and a Python analysis script, and crisis-triage does the same. That per-demo scaffolding should be a feature.Without it, a sweep tells you what happened in the model but not what the LLM said, so there is no way to correlate outcomes with prompts or drift.
What needs to be done
Open questions
Related