feat(skill): add troubleshooting reference; document .openlayerignore and the /v1 asymmetry

.claude-plugin/marketplace.json

@@ -1,6 +1,6 @@
 {
   "name": "openlayer",
-  "version": "0.3.0",
+  "version": "0.4.0",
   "description": "Openlayer agent skills marketplace \u2014 teach AI coding agents to integrate apps with Openlayer.",
   "owner": {
     "name": "Openlayer",

.claude-plugin/plugin.json

@@ -1,7 +1,7 @@
 {
   "name": "openlayer",
   "description": "Skills for working with Openlayer, the AI evaluation and observability platform \u2014 tracing/monitoring, offline testing, evals, and CI/CD gates.",
-  "version": "0.3.0",
+  "version": "0.4.0",
   "author": {
     "name": "Openlayer",
     "email": "support@openlayer.com"

.cursor-plugin/plugin.json

@@ -1,7 +1,7 @@
 {
   "name": "openlayer",
   "displayName": "Openlayer",
-  "version": "0.3.0",
+  "version": "0.4.0",
   "description": "Skills for working with Openlayer \u2014 the AI evaluation and observability platform for tracing/monitoring, offline testing, evals, and CI/CD gates.",
   "author": {
     "name": "Openlayer",

skills/openlayer/SKILL.md

@@ -44,8 +44,12 @@ Follow these for ALL Openlayer work:
      offline before shipping?" Many teams eventually use both.
 4. **Minimal footprint.** Wrap existing clients and decorate existing functions — don't rewrite app logic.
 5. **Never hardcode keys.** Use env vars: `OPENLAYER_API_KEY`, `OPENLAYER_BASE_URL` (self-hosted/local
-   only), `OPENLAYER_INFERENCE_PIPELINE_ID`, `OPENLAYER_DISABLE_PUBLISH`. Don't ask the user to paste
+   only — the SDKs need it to **include** `/v1`, unlike the CLI profile URL; see `references/cli.md`),
+   `OPENLAYER_INFERENCE_PIPELINE_ID`, `OPENLAYER_DISABLE_PUBLISH`. Don't ask the user to paste
    keys into chat — have them set the env var or a `.env`. Keys: Workspace settings → API keys.
+6. **Debugging is not rebuilding.** When something already set up isn't working, diagnose before
+   editing — most failures are an unset env var, a base URL, or an app that never loaded its env file,
+   not wrong instrumentation. Start from `references/troubleshooting.md`.
 
 ## Data access (the "API" plane)
 
@@ -75,6 +79,7 @@ Preference order: search (when topic is fuzzy) → `llms.txt` lookup → fetch t
 
 | If the user wants to…                                   | Read                                      |
 | ------------------------------------------------------- | ----------------------------------------- |
+| **Debug a setup that isn't working** (no traces, push/413, auth, base URL) | `references/troubleshooting.md` |
 | Add live tracing / observability to code               | `references/monitoring-instrumentation.md` |
 | Monitor or evaluate a traditional / tabular ML model (scikit-learn, XGBoost, regression/classification) | `references/traditional-ml.md` |
 | Set up offline evals (`openlayer.json` / `tests.json`)  | `references/development-setup.md`          |

skills/openlayer/references/cli.md

@@ -47,6 +47,18 @@ If you ever see `Error: project id not found. Run 'openlayer link'`, do **not**
 `OPENLAYER_PROJECT_ID` instead. (`OPENLAYER_WORKSPACE_ID` is also read from env if needed; the
 workspace is otherwise derived from the API key.)
 
+### `OPENLAYER_BASE_URL`: the CLI and the SDKs disagree about `/v1`
+
+Only relevant for self-hosted or local backends, but a silent failure when it's wrong:
+
+- **SDKs** (Python/TS, and the app you instrument) treat it as the full API root — **include** `/v1`:
+  `https://openlayer.internal/v1`, `http://localhost:8090/v1`.
+- **CLI** appends `/v1` per request, so its stored profile URL **omits** it. Recent CLI versions
+  normalize either form on input; older ones do not.
+
+When both run in the same shell, set the SDK form (`.../v1`) — the CLI strips a trailing `/v1`, but an
+SDK given the CLI form gets 404s or silently publishes nothing.
+
 ## Core commands
 
 | Command | Use |
@@ -61,6 +73,34 @@ workspace is otherwise derived from the API key.)
 | `openlayer profile` | Manage CLI profiles |
 | `openlayer update` | Update the CLI |
 
+## Controlling what gets uploaded (`.openlayerignore`)
+
+`push` bundles **the entire directory containing `openlayer.json`** and uploads it. A virtualenv,
+`node_modules`, model checkpoints, or dataset caches sitting next to that file are swept in — this is
+the usual cause of a multi-hundred-megabyte upload or a `413`.
+
+Put a `.openlayerignore` next to `openlayer.json`. It uses gitignore syntax:
+
+```gitignore
+node_modules/
+.venv/
+.next/
+data/raw_dumps/
+*.ckpt
+```
+
+Recent CLI versions already exclude common dependency and build directories (`node_modules/`, `.venv/`,
+`.next/`, `dist/`, `build/`, `__pycache__/`, `.git/`, checkpoint files) by default, and warn before
+uploading an oversized bundle. Two consequences worth knowing:
+
+- Re-include a default with a `!` negation (`!dist/`) — needed if your `model.outputDirectory` or
+  metrics directory is named like one of them.
+- On an older CLI, none of this is automatic: list everything explicitly.
+
+Keep whatever the eval actually reads — `openlayer.json`, the runner script, `requirements.txt`, and
+the dataset — and exclude the rest. The remote run reinstalls dependencies from your `installCommand`;
+it never needs your local ones.
+
 ## Common Mistakes
 
 | Mistake | Problem | Fix |
@@ -73,3 +113,6 @@ workspace is otherwise derived from the API key.)
 | Passing `--wait=false` in CI | Job passes before results land | Keep the default (`--wait` is true) and fail on failed tests (`references/ci-cd.md`) |
 | Wrong `export` arg order | Empty/incorrect export | `export <pipelineId> <start> <end>` |
 | Wrong profile/workspace | Pushes to the wrong place | Check `openlayer whoami` / `--profile-name` |
+| `openlayer.json` sits beside `node_modules`/`.venv`/checkpoints | Bundle balloons to hundreds of MB; upload is slow or fails with `413` | Add a `.openlayerignore` next to `openlayer.json` (see above) |
+| `OPENLAYER_BASE_URL` without `/v1` for the SDK | 404s or traces that never land | SDKs need the `/v1`; the CLI profile URL doesn't (see above) |
+| `model.outputDirectory` named `dist`/`build` | Excluded by the CLI's default ignores, so outputs never upload | Rename it, or re-include with `!dist/` in `.openlayerignore` |

skills/openlayer/references/development-setup.md

@@ -85,6 +85,28 @@ openlayer push -m "message"          # waits for results by default (--wait); ad
 and set `OPENLAYER_PROJECT_ID`. If you see `project id not found. Run 'openlayer link'`, set that env
 var instead of running `link`. See `references/cli.md`.
 
+`OPENLAYER_BASE_URL` for the CLI omits the `/v1` that the SDKs require — see `references/cli.md` if
+you're on a self-hosted or local backend.
+
+#### Keeping the bundle small
+
+`push` uploads **everything in the directory containing `openlayer.json`**. When that directory is
+also an application root, the virtualenv, `node_modules`, or model checkpoints go up with it — the
+usual cause of a slow push or a `413`. Write a `.openlayerignore` next to `openlayer.json`:
+
+```gitignore
+node_modules/
+.venv/
+.next/
+*.ckpt
+```
+
+Recent CLI versions exclude common dependency and build directories by default and warn before an
+oversized upload; a `!dist/`-style negation re-includes one if your output or metrics directory
+happens to share a name. On older versions nothing is excluded automatically. Keep what the eval
+reads — the config, the runner, `requirements.txt`, the dataset — and drop the rest; the remote run
+reinstalls dependencies via your `installCommand`. Details in `references/cli.md`.
+
 After push, Openlayer runs the model, generates insights, evaluates tests, and reports pass/fail
 (commit logs in the app, Git, or REST `commits.test_results`). See `references/cli.md` and
 `references/data-access.md`.
@@ -115,4 +137,5 @@ see `references/data-access.md`), understand why, then fix and re-push.
 | Skipping `openlayer validate` | Push fails late with a cryptic error | Always `validate` first |
 | Secrets committed in `openlayer.json` | Leak | Keep keys in env vars, not the config |
 | `modelType: "full"` with no/ wrong `batchCommand` | Output generation fails | Use `{{ path }}`/`{{ name }}` placeholders, or use `"shell"` with precomputed outputs |
-| `push` bundles the whole project dir (incl. `.venv`/`node_modules`) | Upload 413 Request Entity Too Large | Push from a clean dir — keep the virtualenv / large artifacts outside the project root |
+| `push` bundles the whole directory holding `openlayer.json` (incl. `.venv`/`node_modules`) | Hundreds of MB uploaded; `413 Request Entity Too Large` | Add a `.openlayerignore` next to `openlayer.json` — see "Keeping the bundle small" above |
+| `model.outputDirectory` named `dist`/`build`/`target` | Excluded by the CLI's default ignores, so generated outputs never upload | Rename it, or re-include it with `!dist/` in `.openlayerignore` |

skills/openlayer/references/monitoring-instrumentation.md

@@ -73,6 +73,37 @@ def answer(question: str) -> str:
     return llm(question, context)
 ```
 
+#### Every parameter of a traced function becomes a column — keep them serializable
+
+`@trace()` reflects on the decorated function's **declared parameters** and captures each argument as
+an input variable. Those names are written into the row's `inputVariableNames`, but the values are
+JSON-serialized — and anything that doesn't survive `JSON.stringify` / `json.dumps` (a callback, a
+client handle, a DB session, a file object) is dropped from the payload while its **name stays in the
+config**. The row then arrives one column short of what it declared:
+
+```
+400 There is one issue with the row/config streamed: 1. Not all input variables specified in
+`inputVariableNames` are in the dataset.
+```
+
+Fix it by changing the signature, not by removing the trace — pass plain data and derive the
+non-serializable thing inside the function body:
+
+```ts
+// ✗ measureTime is declared, captured, then silently dropped on serialization
+const processQuery = trace(async (messages, model, measureTime: (l: string) => void) => { … })
+
+// ✓ every captured parameter is serializable data
+const processQuery = trace(async (messages, model, apiStart: number) => {
+  const measureTime = (label: string) => console.log(label, Date.now() - apiStart)
+  …
+})
+```
+
+The same applies in Python: a traced function taking `db_session`, `client`, or an `on_progress`
+callback will declare a column it can't fill. If a non-serializable argument genuinely has to stay in
+the signature, wrap the traced work in an inner function that takes only data and decorate that one.
+
 `context_kwarg`/`question_kwarg` are an alternative to `log_context`/`log_question`, but they read from
 the decorated function's **own declared parameters** — `@trace(context_kwarg="context")` errors with
 `Context kwarg 'context' not found in inputs` unless `context` is a parameter of that function. When the
@@ -125,3 +156,5 @@ Tests run on the live traces — to add quality checks, see the tests docs
 | Adding lots of instrumentation before verifying | Hard to debug why nothing lands | Verify one trace publishes first, then enrich |
 | Guessing provider wrapper names/signatures from memory | Wrong import, runtime error | Fetch the current snippet from the instrument / alternative-integrations docs |
 | Assuming TS auto-publishes without the pipeline env var | Silent no-op | TS publishes only when `OPENLAYER_INFERENCE_PIPELINE_ID` is set; `OPENLAYER_DISABLE_PUBLISH=true` disables |
+| A traced function takes a callback, client, or session as a parameter | 400 `Not all input variables specified in inputVariableNames are in the dataset` — the name is declared, the value is dropped on serialization | Pass plain data and build the non-serializable value inside the function (see "keep them serializable" above) |
+| `OPENLAYER_BASE_URL` set without the `/v1` suffix | 404s on every publish, or traces that silently never land | The SDK's base URL is the full API root and **includes** `/v1` — unlike the CLI profile URL, which omits it (see `references/cli.md`) |

skills/openlayer/references/troubleshooting.md

@@ -0,0 +1,98 @@
+---
+name: openlayer-troubleshooting
+description: Diagnose a broken Openlayer setup — traces not appearing, push/bundle failures, auth and base-URL problems, validation errors. Use when something that should be working isn't, or when asked to debug an existing integration rather than build one.
+---
+
+# Openlayer — Troubleshooting
+
+Use this when Openlayer is already set up and something isn't working. `openlayer init` routes its
+"something went wrong" escape hatch here, so the report you're handed is often a user's paste of an
+error with little else.
+
+**Diagnose before you change anything.** The most common wrong turn is rewriting instrumentation that
+was already correct, when the actual problem was an unset env var, a base URL missing `/v1`, or an app
+that never loaded `.env.openlayer`. Confirm where the failure is before editing code.
+
+## Triage order
+
+1. **Reproduce.** Run the thing that fails and read the actual error. Don't work from the user's
+   summary alone — a "traces aren't showing up" report is frequently a 400 or 401 in the app logs.
+2. **Check credentials and target** (below) — this covers most "nothing arrives" cases.
+3. **Confirm the request leaves the app.** Set `OPENLAYER_VERBOSE=true` if available, or log around
+   the publish call. A trace that is never published fails differently from one rejected by the API.
+4. **Only then read the instrumentation** for logic errors.
+
+## Nothing arrives in Openlayer
+
+Check in this order — each is a silent failure, not an exception:
+
+| Check | How | If wrong |
+| --- | --- | --- |
+| `OPENLAYER_API_KEY` set **in the app's process** | Print it (masked) at startup | The app doesn't inherit your shell. `init` writes `.env.openlayer`; the app must load it (dotenv, or your framework's env config) |
+| `OPENLAYER_INFERENCE_PIPELINE_ID` set | Same | Without it traces are created and never published — the single most common cause |
+| `OPENLAYER_BASE_URL` (self-hosted/local only) ends in `/v1` | Print it | SDKs need the full API root **with** `/v1`; the CLI profile URL omits it. See `references/cli.md` |
+| `OPENLAYER_DISABLE_PUBLISH` unset | Same | If truthy, everything is traced and nothing is sent — common leftover from local dev |
+| The traced code path actually ran | Add a log line inside the traced function | Wrapping a client the request path doesn't use traces nothing |
+| The wrapped client is the one used | Grep for other client constructions | A second, unwrapped client elsewhere bypasses tracing |
+
+If all of these are right and traces still don't appear, the request is probably being **rejected**.
+Look for a 4xx in the app's logs and match it below.
+
+## API errors
+
+| Error | Cause | Fix |
+| --- | --- | --- |
+| `400 Not all input variables specified in inputVariableNames are in the dataset` | A traced function declares a parameter whose value isn't JSON-serializable (callback, client, DB session). The name is declared; the value is dropped on serialization | Pass plain data and derive the non-serializable value inside the function — see `references/monitoring-instrumentation.md` |
+| `401` / `403` | Wrong key, or a key from a different workspace | `openlayer whoami`. Keys are per-workspace: Workspace settings → API keys |
+| `404` on every call, self-hosted | Base URL missing `/v1` | See the base-URL row above |
+| `413 Request Entity Too Large` on push | The bundle includes dependency trees or build output | `.openlayerignore` next to `openlayer.json` — see `references/cli.md` |
+| `Context kwarg 'context' not found in inputs` | `@trace(context_kwarg=…)` names something that isn't a parameter of that function | Use `log_context()` / `log_question()` when the step computes its context internally |
+| Row published but a column is empty | Set through the wrong mechanism (e.g. context via `update_current_trace`) | Check the enrichment table in `references/monitoring-instrumentation.md` |
+
+## `openlayer push` and bundling
+
+| Symptom | Cause | Fix |
+| --- | --- | --- |
+| Upload is huge or slow; `413` | `push` bundles the **entire directory containing `openlayer.json`** — a virtualenv, `node_modules`, `.next`, or checkpoints beside it are swept in | Add `.openlayerignore`. Recent CLI versions exclude common dependency/build directories by default and warn before an oversized upload |
+| Generated outputs missing from the run | `model.outputDirectory` matches an ignore pattern (`dist`, `build`, …) | Rename it, or re-include with `!dist/` |
+| `project id not found. Run 'openlayer link'` in an agent/CI | `link` is TTY-only | Set `OPENLAYER_PROJECT_ID`; never run `link` non-interactively |
+| Push fails late with a cryptic error | Skipped validation | `openlayer validate` first |
+| Tests SKIPPED rather than failing | Output column isn't the canonical `openlayer_output` | Align the column name — see `references/development-setup.md` |
+| Commit "passes" before results exist | `--wait=false` | Keep the default `--wait` — see `references/ci-cd.md` |
+
+## Environment and process problems
+
+- **The app doesn't see the env vars.** `.env.openlayer` is a file, not magic: something has to load
+  it. Check for `dotenv`/`python-dotenv` and that it runs *before* the Openlayer import. Next.js and
+  similar frameworks only expose specific env files — verify rather than assume.
+- **Two Python/Node environments.** The SDK installed in one and the app running in another looks
+  exactly like "the SDK doesn't work". Check the interpreter/`node_modules` the app actually uses.
+- **Stale SDK.** Method names and wrapper signatures change; an old version fails in confusing ways.
+  See `references/sdk-upgrade.md`.
+- **Local backend not reachable.** `curl $OPENLAYER_BASE_URL/...` from the same shell before blaming
+  the SDK.
+
+## When the root cause is in the user's app
+
+Say so plainly rather than working around it in the instrumentation. Restructuring a function so its
+traced parameters are serializable is a legitimate fix; disabling tracing, catching and swallowing the
+publish error, or removing columns to make a 400 go away are not — they hide the failure instead of
+fixing it.
+
+## Escalation
+
+If the failure survives all of the above, collect for a bug report: SDK version and language, the
+exact error with its status code, the relevant env vars (**masked**), and whether it reproduces with a
+minimal script. Confirm current behaviour against the docs (`references/docs-access.md`) before
+concluding it's a platform bug — the SDKs change often.
+
+## Common Mistakes
+
+| Mistake | Problem | Fix |
+| ------- | ------- | --- |
+| Rewriting instrumentation before reproducing the failure | Correct code gets churned while the real cause (env var, base URL, unloaded `.env`) survives | Reproduce and read the actual error first |
+| Trusting the user's summary over the logs | "No traces" is often a 400/401 the app swallowed | Find the status code before theorising |
+| Fixing a 400 by deleting the offending column | Hides a real config/row mismatch | Make the value serializable — see `references/monitoring-instrumentation.md` |
+| Wrapping the publish call in try/except to silence it | Failures become invisible instead of fixed | Let it raise until the cause is understood |
+| Assuming the app inherits your shell's env | The process may load nothing at all | Verify from inside the running app |
+| Debugging the SDK when the CLI is what's failing (or vice-versa) | Wrong plane entirely | Dev/push problems → `references/cli.md`; trace problems → `references/monitoring-instrumentation.md` |