Discover examples for remotely-loaded skills

[brandomr]

Problem

A skill loaded from a URL exposes no examples at all. _discover_examples scans the examples/ directory, which only works for a local skill:

if source_type == "local" and base_path:
    ...
# For remote skills, examples must be declared in the frontmatter (not yet implemented).

So a remote skill gives the agent instructions and reference files, and silently drops every worked example the author shipped. That's the tier the agent most wants before writing code — and the loss is invisible. The prompt reports has_code_examples: false, which reads as "this skill has no examples" rather than "this skill's examples could not be seen".

Reproduced against two published skills served from raw.githubusercontent.com:

slug='rosetta'   source=remote  files(5)  examples(0)
slug='deepscale' source=remote  files(5)  examples(0)

Both ship four runnable examples.

Approach

Take examples from the body rather than from the filesystem.

Skill authors already publish an example listing as markdown links — the convention across skills I looked at is

- [examples/basic_fetch.py](examples/basic_fetch.py) — obs + hindcast fetch, save to NetCDF

and _build_skill_integration already has the parsed body in hand. A new extract_example_references() pulls (path, title, description) out of those links: titles from the link text (falling back to the filename when the author wrote the path as the text), descriptions from the text following the link.

Notably this costs no extra network requests — discovery stays free and content is still fetched on demand at tier 3. Declaring examples in frontmatter, the originally-sketched approach, would either need one request per example up front or duplicate every title and description into the frontmatter.

Local skills keep scanning the directory, which stays authoritative. This is additive: a remote skill with no example links behaves exactly as before.

Result

Same two skills, after:

== rosetta: 4 examples
   basic_fetch.py            desc='obs + hindcast fetch, save to NetCDF/GeoTIFF'
   multi_model_assemble.py   desc='`assemble()` roster → deepscale-ready arrays'
   shapefile_region.py       desc='country clipping, center vs cover'
   s2s_fetch.py              desc='sub-seasonal issuance + on-the-fly reforecasts'
   -> fetch examples/basic_fetch.py: 1916 chars OK
== deepscale: 4 examples
   end_to_end_forecast.py    desc='optimize → forecast → honest CV verification → PDF report'
   seasonal_mme_pipeline.py  desc='one-call multi-model MME with `seasonal_mme`'
   calibration.py            desc='eReg multi-model calibration + logistic/WVG index calibration'
   ensemble_and_reporting.py desc='strategies, safeguards, skill comparison, plots'
   -> fetch examples/end_to_end_forecast.py: 3029 chars OK

Tests

15 new tests covering the extractor (listing form, descriptive link text, separator stripping, dedup, nested paths, non-example links ignored) and end-to-end remote discovery (descriptions preserved, content not fetched during discovery, exactly one request made, examples not double-counted as file resources, skills without example links unaffected).

Full suite: 489 passed.

🤖 Generated with Claude Code

https://claude.ai/code/session_01555XSwMPRxBZFAEsvubUMz

[mattprintz]

Good catch on a legitimate oversight. Suggested a potential different path below that I think is workable but would be cleaner and more maintainable overall.

[mattprintz]

Instead of manually extracting the examples separately, I think a better solution might be to remove the guard clause in the extract_file_references function above at lines 158-159.
This would keep the extraction consistent and that function is already being called and should handle this situation, I believe.

Doing so will likely mean that we have to dedupe examples detected via local path traversal (_discover_examples class method below, starting at line 507), but I think that should not be difficult.

[brandomr]

Good call — done in c8fe864. extract_example_references is gone.

Guard dropped from extract_file_references. _build_skill_integration routes examples/ paths to SkillExampleResource and the rest to SkillFileResource. _discover_examples merges the referenced paths with the directory scan on filename, scan winning since it has a real parsed title.

Two things your version catches that mine didn't:

• Local skills had the same bug. The scan globs *.md, so examples/*.py was invisible locally too, not just remotely.
• A bare examples/ link would have become a SkillFileResource that 404s once the guard came off. Dropped in the extractor now.

One loss worth flagging: I was scraping each link's trailing text into description. That's gone. Seems fine to me — load_skill_instructions returns the whole body including the author's example listing, and the agent has to call that before it can ask for an example by name, so the descriptions were already in context. Say the word if you'd rather keep them.

Tests: dropped TestExtractExampleReferences, inverted test_examples_paths_excluded, added TestExampleSourcesAreDeduped for the merge. 487 pass. Re-checked against the two published skills — 4 examples each, nothing leaking into file resources.

[mattprintz]

Fix looks good. Tested with the dump command and no issues.