diff --git a/docker/docker-compose.yml b/docker/docker-compose.yml index 754662d..156f1c2 100644 --- a/docker/docker-compose.yml +++ b/docker/docker-compose.yml @@ -209,7 +209,11 @@ services: langfuse-redis: { condition: service_healthy } langfuse-minio: { condition: service_healthy } healthcheck: - test: ["CMD", "wget", "--no-verbose", "--tries=1", "--spider", "http://localhost:3000/api/public/health"] + # langfuse-web (Next.js) binds the container's own IP, NOT loopback, so a + # localhost probe is refused and the container shows perpetually "unhealthy" + # even while it serves fine (host :3100 returns 200 throughout). Probe the + # container IP via $(hostname -i) instead. + test: ["CMD-SHELL", "wget --no-verbose --tries=1 --spider http://$(hostname -i):3000/api/public/health || exit 1"] interval: 30s timeout: 10s retries: 5 diff --git a/tasks/VERIFY-WHEN-DATA-ACCRUES.md b/tasks/VERIFY-WHEN-DATA-ACCRUES.md index b23f1f0..5d0024f 100644 --- a/tasks/VERIFY-WHEN-DATA-ACCRUES.md +++ b/tasks/VERIFY-WHEN-DATA-ACCRUES.md @@ -27,6 +27,18 @@ Most of these are testable *today* with synthetic data — prefer this over wait ## Open (mechanism proven by stub; live demonstration pending) +> **2026-07-30 verification pass (live data checked).** Langfuse `devflow-skills`: 81 traces, +> 3/81 attributed (skill_name 2, slash 1), and only 2 scores (both `trace-review` tessl_review +> dated 2026-07-06). Sidecar `~/.devflow/skill-activations.jsonl`: 113 rows / 59 sessions / 24 +> days, but only 3 of those sessions join a Langfuse trace `sessionId`. Two INFRA faults were +> found and fixed this pass: `langfuse-web` showed perpetually "unhealthy" (healthcheck probed +> `localhost:3000` but the app binds the container IP; fixed to `$(hostname -i)`), and a +> redundant bundled-hindsight container was crash-looping (native daemon owns :8888; orphan +> removed). The score-push path was then re-verified end to end (a `tessl-push` landed in +> ClickHouse in ~5s). Net: the accrual items below are still blocked on real *volume*, not on +> a broken pipeline. `0 scores since 07-06` = no create/optimize-skill reviews ran (+ web +> degraded for part of the window), not a dropped-push bug. + - [ ] **Score-DOWN flag on real accumulated data.** Stub: bats `_flag_regressions` + the two-window eval-score test. Live: once >=2 weeks of pushed scores exist for a skill, confirm a real quality drop shows `🟠/🔴 score -X`. @@ -39,11 +51,9 @@ Most of these are testable *today* with synthetic data — prefer this over wait is installed and confirm `hook` appears in the Attribution breakdown and coverage rises above the retroactive 3/57. Trigger: 1 week of real sessions after `devflow init` installs the PreToolUse:Skill hook AND telemetry env is on. -- [ ] **Enrichment hook writing rows in real sessions.** Stub: bats feeds a synthetic - PreToolUse payload and asserts the sidecar row. Live: after `devflow init`, invoke any - skill and confirm a real row lands in `~/.devflow/skill-activations.jsonl` with a - `session_id` that matches a Langfuse trace `sessionId`. Trigger: first real skill run - post-install. +- [x] **Enrichment hook writing rows in real sessions.** — VERIFIED (see Verified below). + The `session_id`-matches-a-trace half is tracked under "Rung-3 sidecar coverage" (only 3/59 + join today; needs sessions with telemetry + hook both on). - [ ] **Auto-feeder populating the score column unattended.** Stub: manual `tessl-push` e2e proved display. Live: confirm that after a skill review runs (create-skill / optimize-skill / pre-PR gate), the score lands without a manual push. Trigger: first @@ -51,4 +61,12 @@ Most of these are testable *today* with synthetic data — prefer this over wait ## Verified -(move items here with date + evidence once the live check passes) +- [x] **Enrichment hook writing rows in real sessions** — 2026-07-30. `~/.devflow/skill-activations.jsonl` + has 113 rows / 59 distinct sessions spanning 2026-07-06 → 2026-07-30, each a valid + `{session_id, ts, skill}` (e.g. `devflow:resolve-repo`, `superpowers:executing-plans`). The hook + fires on real skill runs post-install. (The downstream `session_id` → trace `sessionId` join is a + separate accrual item, "Rung-3 sidecar coverage".) +- [x] **Score-push path lands a score end to end** — 2026-07-30. `eval/lib/tessl-push.sh trace-review 88` + ingested and appeared in Langfuse (scores 2 → 3) within ~5s, after fixing the `langfuse-web` + healthcheck. Confirms the auto-feeder → Langfuse → ClickHouse pipe works; the score *column* + filling unattended still needs a real create/optimize-skill review to run (accrual item above).