Skip to content

Commit a2f61c2

Browse files
docs(observability): document the blank otel-collector auth failure mode (#90)
Found live 2026-07-30: docker/.env's LANGFUSE_OTEL_BASIC_AUTH had gone empty (cause unknown - predates this session's changes), so otel-collector sent Authorization: Basic <nothing> to langfuse-web, which rejected every batch with HTTP 401. This silently dropped ALL Claude Code telemetry for roughly 27 days (last real claude_code.tool observation before the fix: 2026-07-03) with zero visible error - devflow trace-review run just quietly reported 0 observations / 0 attribution, and nothing else surfaces this class of failure. Root-caused and fixed live (recomputed the value from ~/.config/zsh/secrets, docker compose up -d --force-recreate otel-collector - a plain restart does not re-read a changed .env), verified by a fresh claude_code.tool observation landing seconds later and devflow trace-review run immediately showing observations=24, attribution 1/29. Documented as gotcha #4 alongside the existing three, matching the file's own convention, so this is a 30-second diagnosis next time instead of the investigation it took here.
1 parent c5962b5 commit a2f61c2

1 file changed

Lines changed: 14 additions & 0 deletions

File tree

‎eval/OBSERVABILITY.md‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -81,6 +81,20 @@ to **`langfuse/langfuse:3.200.0`** (web + worker, identical).
8181
`No such container <hash>_devflow-langfuse-web`, leaving the OLD container serving
8282
stale env. Fix: a **full `down` then fresh `up`** (a fresh CREATE has no rename), not
8383
a partial recreate.
84+
4. **A blank `LANGFUSE_OTEL_BASIC_AUTH` in `docker/.env` fails SILENTLY and drops ALL
85+
Claude Code telemetry** (found 2026-07-30, after ~27 days of missing traces went
86+
unnoticed). If that var is unset/empty, `otel-collector` sends `Authorization: Basic `
87+
(nothing after it) to `langfuse-web`, which rejects every batch with HTTP 401 -
88+
`otel-collector` logs `Exporting failed. Dropping data.` and the CLI-facing symptom is
89+
`devflow trace-review run` quietly showing `"observations": 0` and zero attribution,
90+
with NO error surfaced anywhere the user would see. Diagnose: `docker logs
91+
devflow-langfuse-otel-collector-1 --since 1h | grep 401`. Fix: recompute the value
92+
(`base64("$LANGFUSE_PUBLIC_KEY:$LANGFUSE_SECRET_KEY")`, sourcing keys from
93+
`~/.config/zsh/secrets`) into `docker/.env`, then `docker compose up -d
94+
--force-recreate otel-collector` (env vars are baked at container creation - a plain
95+
`restart` does NOT re-read a changed `.env`). Verify by re-querying
96+
`/api/public/observations?name=claude_code.tool&orderBy=startTime.desc` for a
97+
just-now timestamp, not just by the absence of new 401s.
8498

8599
### Score-push bridge (feeds `devflow trace-review`'s score column)
86100
Push a skill's quality score as a `devflow-eval` trace so trace-review can key it by

0 commit comments

Comments
 (0)