Observer can't process backlogs past a few MB (cat-based read doesn't scale)

[proofoftom]

Summary

The observer agent's documented workflow (agents/observer.md) says:

1. Read observations: cat .claude/homunculus/observations.jsonl

This works for fresh sessions but breaks down once a real backlog accumulates. After 3 months of normal use on a Drupal project, my observations.jsonl was 53 MB / 40,121 lines / ~13M tokens — far past any single agent's context window (haiku 4.5 = 200k). A naive observer run would either fail on context overflow or silently never produce instincts.

Symptoms

• Plugin reports instincts.personal: 0 despite months of activity
• No sessions/ entries despite the SessionStart hook firing
• observations.jsonl grows unbounded

This is what I assume is happening: observer spawns, tries cat, hits context limits, fails, no error surfaces. Instincts directory stays empty.

Repro

1. Use a homunculus-enabled project actively for ~2-3 months (or seed with a synthetic 30k-line observations.jsonl)
2. Check .claude/homunculus/instincts/personal/ — empty
3. Check size of observations.jsonl — likely tens of MB

Workaround

I distilled my 40k backlog manually:

1. Archive raw file: cp observations.jsonl observations.archive.jsonl
2. Pre-filter to high-signal events with a Python script: keep all prompt events, tool calls within 3 calls after each prompt, tool errors. Reduced 40,123 → 4,930 events (~5MB).
3. Chunk into 10 equal parts (~500 events each, ~150k tokens — fits haiku)
4. Spawn 10 observer agents in parallel (background), each pointed at its own chunk via prompt override
5. Dedupe instincts and update identity.json after all complete

Result: 80 instincts across 7 clustered domains. Cost ~$0.50 in haiku across 11 background agents. (1 retry — see related cwd issue #10.)

Suggested fix

Build chunked/streaming processing into the observer itself:

1. Auto-archive on threshold — when observations.jsonl exceeds some size (say 5MB / 5k lines), the observer (or a SessionStart hook) auto-archives and clears, processing the archive in chunks.
2. Pre-filter pass — observer reads the file in two passes: first a cheap line-by-line filter (Python/jq) to extract high-signal events (prompts + errors + windows around prompts), then the LLM-powered pattern analysis runs on the filter output.
3. Streaming chunked spawn — observer detects size and spawns sub-observer agents per N-line chunk, then a roll-up agent dedupes their findings.

(2) is probably the biggest leverage — most observation lines are repetitive Read/Bash/Grep with low pattern-density. Filtering before LLM analysis cuts cost ~10x.

Context

• Plugin: 2.0.0-alpha
• Filesystem: .claude/homunculus/observations.jsonl ~ 53 MB after 90 days
• Project: large Drupal codebase, heavy CLI use