Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion docker/docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -209,7 +209,11 @@ services:
langfuse-redis: { condition: service_healthy }
langfuse-minio: { condition: service_healthy }
healthcheck:
test: ["CMD", "wget", "--no-verbose", "--tries=1", "--spider", "http://localhost:3000/api/public/health"]
# langfuse-web (Next.js) binds the container's own IP, NOT loopback, so a
# localhost probe is refused and the container shows perpetually "unhealthy"
# even while it serves fine (host :3100 returns 200 throughout). Probe the
# container IP via $(hostname -i) instead.
test: ["CMD-SHELL", "wget --no-verbose --tries=1 --spider http://$(hostname -i):3000/api/public/health || exit 1"]
interval: 30s
timeout: 10s
retries: 5
Expand Down
30 changes: 24 additions & 6 deletions tasks/VERIFY-WHEN-DATA-ACCRUES.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,18 @@ Most of these are testable *today* with synthetic data — prefer this over wait

## Open (mechanism proven by stub; live demonstration pending)

> **2026-07-30 verification pass (live data checked).** Langfuse `devflow-skills`: 81 traces,
> 3/81 attributed (skill_name 2, slash 1), and only 2 scores (both `trace-review` tessl_review
> dated 2026-07-06). Sidecar `~/.devflow/skill-activations.jsonl`: 113 rows / 59 sessions / 24
> days, but only 3 of those sessions join a Langfuse trace `sessionId`. Two INFRA faults were
> found and fixed this pass: `langfuse-web` showed perpetually "unhealthy" (healthcheck probed
> `localhost:3000` but the app binds the container IP; fixed to `$(hostname -i)`), and a
> redundant bundled-hindsight container was crash-looping (native daemon owns :8888; orphan
> removed). The score-push path was then re-verified end to end (a `tessl-push` landed in
> ClickHouse in ~5s). Net: the accrual items below are still blocked on real *volume*, not on
> a broken pipeline. `0 scores since 07-06` = no create/optimize-skill reviews ran (+ web
> degraded for part of the window), not a dropped-push bug.

- [ ] **Score-DOWN flag on real accumulated data.** Stub: bats
`_flag_regressions` + the two-window eval-score test. Live: once >=2 weeks of pushed
scores exist for a skill, confirm a real quality drop shows `🟠/🔴 score -X`.
Expand All @@ -39,16 +51,22 @@ Most of these are testable *today* with synthetic data — prefer this over wait
is installed and confirm `hook` appears in the Attribution breakdown and coverage rises
above the retroactive 3/57. Trigger: 1 week of real sessions after `devflow init`
installs the PreToolUse:Skill hook AND telemetry env is on.
- [ ] **Enrichment hook writing rows in real sessions.** Stub: bats feeds a synthetic
PreToolUse payload and asserts the sidecar row. Live: after `devflow init`, invoke any
skill and confirm a real row lands in `~/.devflow/skill-activations.jsonl` with a
`session_id` that matches a Langfuse trace `sessionId`. Trigger: first real skill run
post-install.
- [x] **Enrichment hook writing rows in real sessions.** — VERIFIED (see Verified below).
The `session_id`-matches-a-trace half is tracked under "Rung-3 sidecar coverage" (only 3/59
join today; needs sessions with telemetry + hook both on).
- [ ] **Auto-feeder populating the score column unattended.** Stub: manual `tessl-push`
e2e proved display. Live: confirm that after a skill review runs (create-skill /
optimize-skill / pre-PR gate), the score lands without a manual push. Trigger: first
real skill-review after the feeder is wired.

## Verified

(move items here with date + evidence once the live check passes)
- [x] **Enrichment hook writing rows in real sessions** — 2026-07-30. `~/.devflow/skill-activations.jsonl`
has 113 rows / 59 distinct sessions spanning 2026-07-06 → 2026-07-30, each a valid
`{session_id, ts, skill}` (e.g. `devflow:resolve-repo`, `superpowers:executing-plans`). The hook
fires on real skill runs post-install. (The downstream `session_id` → trace `sessionId` join is a
separate accrual item, "Rung-3 sidecar coverage".)
- [x] **Score-push path lands a score end to end** — 2026-07-30. `eval/lib/tessl-push.sh trace-review 88`
ingested and appeared in Langfuse (scores 2 → 3) within ~5s, after fixing the `langfuse-web`
healthcheck. Confirms the auto-feeder → Langfuse → ClickHouse pipe works; the score *column*
filling unattended still needs a real create/optimize-skill review to run (accrual item above).
Loading