Product analysis (PRD, PM, UX/UI, algorithm) + landing conversion edits

docs/product/EVALUATION-ALGORITHM-ANALYSIS.md

@@ -0,0 +1,142 @@
+# Proprietary Evaluation & Skill-Upscaling Algorithm — Analysis & Improvement Plan
+
+> Goal: understand the current scoring/evolution stack end-to-end and propose changes that make it **defensible, harder to game, and genuinely unique** — because in a world of commodity models, *the verifiable quality signal is the moat.*
+
+---
+
+## 1. How it works today
+
+The stack has **four layers**:
+
+### 1.1 Adversarial harness (`src/lib/adversarial/scorer.ts`, `scripts/eval-adversarial.mjs`)
+- Each case declares expectations: `must_refuse`, `must_include[]`, `must_not_include[]`.
+- Pass/fail is decided by **regex refusal detection** (EN/PT/ES) + **case-insensitive substring** checks (`evaluateCase`).
+- Severity weights: `low:1, medium:2, high:4, critical:8` (`scorer.ts:25`).
+- `severity_weighted_score = Σ(weight of passed cases) / Σ(weight of all cases)`.
+
+### 1.2 Evaluator pipeline (`src/lib/skills/pipelines.server.ts`)
+- Scores six axes: **precision, health, safety, hallucination, trigger-rate, efficiency**.
+- **Type-aware weighted blend** (weights sum to 1.0; `pipelines.server.ts:803`):
+  - skill: precision .34 / health .16 / safety .22 / halluc .13 / trigger .09 / eff .06
+  - playbook: precision .38 / health .20 / safety .18 / …
+  - soul: precision .24 / **health .34** / safety .20 / …
+  - guardrail: precision .10 / health .24 / **safety .54** / …
+- Trigger-rate target: **≥80–90% on relevant probes, ≤20% false-positive on irrelevant** (Anthropic-style).
+- Unmeasured axes (trigger/efficiency) have their weight **redistributed proportionally** so a missing measurement neither helps nor hurts.
+
+### 1.3 Public Trust Score (`supabase/.../20260513060000_refresh_cron.sql` → `recompute_trust_scores`, daily cron)
+Bounded `[0,1]`:
+```
+0.10  schema_valid (always 1)
+0.20  adversarial_pass_rate         (avg over runs)
+0.25  adversarial_severity_weighted (avg)
+0.20  real_world_success_rate       (30d: successes/runs)
+0.10  min(1, signed_releases / 3)
+0.05  min(1, ln(age_days + 1) / ln(316))   # ~316d ⇒ full credit
+```
+**Missing components default to 0.5.**
+
+### 1.4 SkillForge evolution loop (`src/lib/skills/forge-loop.functions.ts`, `autolearn` in `pipelines.server.ts`)
+- Auto-learn: root-cause from learnings + failed evals + feedback → cluster → propose smallest coherent patch.
+- **Feedback weighting:** LLM/agent-authored feedback ×2.0 vs human UI feedback ×1.0 (`summarizeFeedback`, `pipelines.server.ts:875`).
+- **Hill-climbing with a no-regression gate** over epochs: candidate accepted only if `candScore > best.score` on a golden mini-batch and no regression (`forge-loop.functions.ts:240`).
+
+This is already a **strong, differentiated** system. The improvements below harden it against the two existential risks: **gaming** and **stale/uninformative signals**.
+
+---
+
+## 2. Weaknesses & risks
+
+### W1. Substring/regex grading is brittle and gameable
+Pass/fail hinges on literal substrings and refusal regexes (`scorer.ts`). Failure modes:
+- **False pass:** a skill can include a `must_include` token while still being unsafe; an attacker-author can pattern-match the harness.
+- **False fail:** a correct refusal phrased outside the EN/PT/ES regex set scores as a failure.
+- **No semantic understanding** of whether the output actually did the harmful thing.
+
+### W2. The 0.5 default rewards absence of evidence
+A brand-new package with **no adversarial runs and no telemetry** inherits `0.5` on the three biggest components (0.20+0.25+0.20 = 65% of the score). A package nobody has stress-tested can show a "decent" Trust Score. This inverts the intended signal.
+
+### W3. No recency weighting on real-world success
+`real_world_success_rate` is a flat 30d average. A skill that broke yesterday on a new model still looks healthy; a skill steadily improving isn't rewarded for trajectory.
+
+### W4. Averaging across runs hides variance / sample size
+Both `adversarial_pass_rate` and `real_world` are plain averages with **no confidence interval**. 2 runs at 100% outscore 10,000 runs at 98% — the opposite of what a security reviewer wants.
+
+### W5. Age component is weak and slightly perverse
+`0.05 × ln(age)` rewards *old* packages even if unmaintained, and gives nothing for *active maintenance / re-attestation freshness*.
+
+### W6. Adversarial suite is static per vertical
+Cases are curated and synced (`sync-adversarial-cases.mjs`). Once an author can see/infer the suite, robustness is overfit to known cases. No adaptive/holdout discipline.
+
+### W7. SkillForge gate optimizes the same metric it reports
+The loop accepts patches that raise `overall_score` on a golden mini-batch — risk of **overfitting to the golden set** (Goodhart). The golden calibration migration exists, but holdout separation should be explicit.
+
+### W8. Single composite number hides the shape
+One `[0,1]` Trust Score collapses safety, competence, and freshness. A guardrail at 0.9 and a skill at 0.9 mean very different things to a buyer.
+
+---
+
+## 3. Improvement plan — making it *unique*
+
+### Trust Score v2 (the headline)
+
+1. **Evidence-gated, not default-0.5.** Replace defaults with an explicit **confidence factor**. Score = `quality × confidence`, where confidence rises with sample size and adversarial coverage. An untested package shows **"Unverified"**, not 0.65. This alone makes the score honest and far harder to game.
+
+2. **Wilson lower-bound instead of raw averages** (W4). For pass rates and success rates, report the **lower bound of a binomial confidence interval** at the package's sample size. "Survived 10k runs at 98%" beats "2 runs at 100%" — exactly the security-reviewer intuition. This is a genuinely differentiating, citable methodology.
+
+3. **Recency-weighted, model-aware success** (W3). Exponential time-decay on executions, and **per-model** success breakdowns so the score reflects "works on the model I actually run." Surface "last verified against {model} on {date}."
+
+4. **Freshness/maintenance term replaces raw age** (W5): reward *recent re-attestation* and *recent passing runs*, decay if a package goes stale or a depended-on model changes.
+
+5. **Multi-dimensional badge, not one number** (W8): publish a **vector** — Safety, Competence, Freshness, Coverage — each with a confidence band, and let the single number be a transparent roll-up. Security buyers can gate on Safety alone.
+
+### Harden the grader (W1)
+
+6. **LLM-judge + rubric ensemble alongside substring checks.** Keep the cheap deterministic checks as a fast gate, add a calibrated **judge model with a rubric** and **golden human labels** (the calibration table already exists) to catch semantic failures. Report judge–human agreement (κ) publicly — that *is* the moat.
+
+7. **Adaptive / holdout adversarial sets** (W6, W7): split cases into a **public training subset** and a **rotating private holdout** the author never sees. Score and rank on the holdout. Add a **community red-team pipeline** (see PM #5) that continuously injects fresh attacks, each credited `SAS-YYYY-NNNN`. A living, adversary-sourced suite is something no static prompt library can copy.
+
+### Make evolution provably better (W7)
+
+8. **Holdout-gated SkillForge.** The loop must accept patches based on a **holdout** score, not the set it learned from, and log the before/after on **both** train and holdout to detect overfitting. Keep the no-regression gate but add a **safety-non-regression hard constraint** (never ship a patch that lowers Safety even if overall rises).
+
+9. **Counterfactual A/B in production** for high-traffic skills: shadow-run the candidate against real traffic before promotion; promote on **statistically significant** improvement, not a mini-batch.
+
+### Transparency = defensibility
+
+10. **Publish the methodology + a reproducible scorer.** The Trust Score is only a moat if buyers believe it. Ship a versioned spec (`trust-score-v2.md`), a reproducible CLI that recomputes a package's score from its public evidence, and signed attestations binding *score version + inputs + result*. "Verify our score yourself, offline" is the unique enterprise pitch.
+
+## 4. Suggested sequencing
+
+| Phase | Change | Why first |
+|---|---|---|
+| 1 | Wilson lower-bound + confidence factor + "Unverified" state (#1,#2,#4) | Cheap, removes the worst gaming/credibility hole |
+| 1 | Fix 0.5 defaults (W2) | Same change set |
+| 2 | Recency/model-aware + freshness term (#3,#4 freshness) | Needs telemetry volume |
+| 2 | LLM-judge ensemble + κ reporting (#6) | Biggest grader quality jump |
+| 3 | Holdout sets + community red-team + holdout-gated Forge (#7,#8) | Structural, highest defensibility |
+| 3 | Multi-dimensional badge + published reproducible spec (#5,#10) | Marketing + enterprise moat |
+
+## 4b. What shipped in this PR
+
+A reproducible, tested **Trust Score v2 core**:
+
+- `src/lib/trust/scoring.ts` — pure, offline-reproducible implementation:
+  Wilson lower-bound (#2), evidence-gating via a confidence factor with an
+  explicit `verified` flag instead of 0.5 defaults (#1, W2), a freshness term
+  replacing raw age (#4, W5), and a multi-dimensional vector — safety /
+  competence / freshness / coverage (#5, W8). Covered by `tests/trust-scoring.test.mjs`.
+- `supabase/migrations/20260529120000_trust_score_v2.sql` — `recompute_trust_scores_v2()`
+  mirrors the TS logic in pure SQL (Wilson + confidence + freshness + dimensions),
+  adds transparency columns, and repoints the nightly cron. v1 is left intact for rollback.
+- `src/lib/adversarial/holdout.ts` — deterministic, salt-rotatable train/holdout
+  split so robustness reflects generalization, not overfitting to known cases
+  (#7, W6). Covered by `tests/adversarial-holdout.test.mjs`.
+
+**Still follow-up (larger bets):** LLM-judge + κ reporting (#6), community red-team
+pipeline + holdout-gated SkillForge promotion (#7, #8), production counterfactual
+A/B (#9), and the published reproducible-spec + signed-methodology surface (#10).
+
+## 5. One-line takeaway
+
+Today the algorithm is a solid weighted average over real signals. To make it **unique and uncopyable**, move from *averages over a static, substring-graded suite* to **confidence-bounded, recency- and model-aware scores over an adversary-sourced, holdout-protected, semantically-judged suite — with a published, reproducible, signed methodology.** That combination is something a prompt library on GitHub structurally cannot match.

docs/product/PM-FEATURE-ANALYSIS.md

@@ -0,0 +1,64 @@
+# Product Manager Analysis — Feature Opportunities
+
+> Companion to `PRD.md`. Framing: what to build next to (a) raise activation, (b) deepen the trust moat, (c) accelerate the flywheel, (d) grow ARPU. Prioritized with a rough **RICE** lens (Reach · Impact · Confidence · Effort).
+
+---
+
+## 1. Where the product is strong today
+
+- **Unique trust stack** — adversarial harness + Ed25519 signing + public Trust Score + per-package offline attestation. No prompt-library competitor ships verifiable proof.
+- **Self-improving supply** — SkillForge auto-learn loop (re-score → patch → A/B gate) means the catalog gets sharper without manual curation.
+- **Distribution** — one MCP endpoint across Claude/Cursor/ChatGPT/Continue/Cline + `npx` install.
+- **Four clean primitives** (skills/playbooks/souls/guardrails) compose well.
+- **Monetization scaffolding** already present: subscriptions, credits, referrals, lineage revenue share, bounties.
+
+## 2. Gaps / opportunities
+
+### A. Activation & onboarding (highest leverage on revenue)
+1. **Guided "Connect → first win" wizard** *(R:H I:H C:H E:M)* — detect the user's tool, give copy-paste config, then auto-suggest 3 skills for their stated role and run a live demo. Today connection + value discovery are separate pages.
+2. **"Try before connect" sandbox** *(R:H I:M C:M E:M)* — run a skill on sample input in-browser (the `/play` and `/run` routes hint at this) so a cold visitor feels the value with zero setup.
+3. **Role/industry-based starter bundles** *(R:M I:H C:H E:S)* — "I run a cardiology clinic" → one-click install a curated pack. Leverages existing `packs` + `collections`.
+
+### B. Trust moat deepening (defensibility)
+4. **Trust Score v2** *(R:M I:H C:M E:M)* — anti-gaming, recency-weighted, confidence intervals, per-attack-class transparency. (Full design in `EVALUATION-ALGORITHM-ANALYSIS.md`.)
+5. **Continuous red-team marketplace** *(R:M I:H C:M E:L)* — bounties where security researchers submit adversarial cases; accepted cases enter the shared suite and earn CVE-style credit (`SAS-YYYY-NNNN`). Turns the community into the moat.
+6. **Compliance export pack** *(R:M I:H C:M E:M)* — one-click "evidence bundle" (signatures + attestation + adversarial breakdown + audit log) formatted for SOC 2 / vendor security reviews. Direct enterprise ACV driver.
+7. **Live drift alerts** *(R:M I:M C:M E:M)* — notify subscribers when a skill they depend on drops below its trust threshold or a new model breaks it.
+
+### C. Flywheel acceleration (supply + telemetry)
+8. **Execution telemetry SDK / opt-in** *(R:H I:H C:M E:M)* — the Trust Score's real-world success component is only as good as reported executions. Make `report-execution` frictionless and incentivized (credits for reporting).
+9. **Author studio** *(R:M I:H C:M E:M)* — analytics dashboard for authors: installs, success rate, revenue, fork lineage, SkillForge patch suggestions to accept/reject. Drives author retention.
+10. **Skill composition / "recipes"** *(R:M I:M C:M E:M)* — let users save a working stack (skills + soul + guardrails) as a shareable, installable recipe. Compounds the `collections` concept.
+
+### D. ARPU / expansion
+11. **Team workspaces + private registry** *(R:M I:H C:M E:L)* — seats, shared installs, private packs, SSO. The natural Enterprise wedge.
+12. **Usage-based credits for Forge generation** *(R:M I:M C:H E:S)* — already partially present; make the "describe → custom skill from your data" path a metered premium feature.
+13. **Verified vendor program** *(R:S I:M C:M E:M)* — paid "verified publisher" badge for orgs; trust + monetization.
+
+### E. Distribution / virality
+14. **Embeddable Trust Badge + leaderboard widgets** *(R:H I:M C:H E:S)* — SVG badge already planned; ship it everywhere (READMEs, docs) as a growth loop.
+15. **IDE-native extensions** *(R:H I:M C:M E:L)* — beyond MCP, a Cursor/VS Code panel for one-click install raises reach.
+
+## 3. Prioritized sequencing (suggested)
+
+**Now (next 1–2 sprints):** 1 (guided wizard), 3 (starter bundles), 8 (telemetry frictionless), 14 (trust badge) — cheap, compounding, activation+flywheel.
+
+**Next:** 2 (sandbox), 4 (Trust Score v2), 6 (compliance export), 9 (author studio).
+
+**Later (bigger bets):** 5 (red-team marketplace), 11 (team workspaces/private registry), 15 (IDE extensions).
+
+## 4. Metrics each bet should move
+
+| Feature | Primary metric |
+|---|---|
+| Guided wizard / sandbox / bundles | Activation %, TTFI |
+| Telemetry / author studio | Executions reported/wk, author retention |
+| Trust Score v2 / compliance export / red-team | Enterprise win rate, trust-badge embeds |
+| Team workspaces / Forge credits | ARPU, free→paid conversion |
+| Trust badge / IDE ext | New visitors, K-factor |
+
+## 5. Open questions for the team
+
+- What is the current activation rate and free→paid conversion baseline? (Wire PostHog funnels first — see PRD §10.)
+- Is the real-world-success telemetry actually populated, or defaulting to 0.5 for most packages? This decides whether to prioritize #8 urgently.
+- Enterprise pipeline: is compliance-export the #1 blocker in sales calls? If so, #6 jumps the queue.