SkillRouteLab

SkillRouteLab is a local-first routing benchmark, catalog auditor, portability checker, context-footprint analyzer, and regression lab for Agent Skills repositories.

Version: 1.0.0

The Agent Skills ecosystem is moving quickly. Skills are now supported across multiple agent runtimes, and the open format is intentionally lightweight: a skill folder contains SKILL.md, optionally with scripts, references, and assets.

That simplicity creates a second-order engineering problem once catalogs grow:

A skill can be valid and still be difficult to discover, ambiguous beside another skill, expensive to activate, unsafe to distribute, or incompatible with a target client.

SkillRouteLab focuses on those catalog-level and routing-level problems.

It is intentionally not a replacement for the official skills-ref validate syntax validator.

What SkillRouteLab does

1. Catalog quality audit

Scans one skill or a complete repository and checks:

• required name
• required description
• name syntax
• 64-character name limit
• directory/name consistency
• 1024-character description limit
• compatibility length
• metadata portability
• allowed-tools syntax
• empty instruction bodies
• local reference integrity
• references escaping the skill root
• deep reference chains
• excessive SKILL.md length
• excessive activation-token footprint
• progressive-disclosure opportunities
• duplicate skill names across a catalog

2. Discovery-quality analysis

A valid skill can still under-trigger.

SkillRouteLab looks for descriptions that:

• are too short
• do not clearly explain when the skill should activate
• overlap heavily with other skills
• create ambiguous routing surfaces

This matters because Agent Skills use progressive disclosure: at startup, agents typically see only skill metadata, and the full instructions are loaded only after a skill is selected.

3. Skill collision detection

SkillRouteLab computes TF-IDF cosine similarity across skill names and descriptions.

Example:

code-review       <-> secure-code-review      0.82
pdf-processing    <-> document-processing     0.66

High similarity is not automatically wrong. It means the catalog needs clearer routing boundaries.

4. Routing benchmark

Create JSONL eval cases:

{"id":"review-pr","query":"Review this pull request for bugs and missing tests","expected_skill":"code-review"}
{"id":"merge-pdf","query":"Merge these PDF files in order","expected_skill":"pdf-processing"}

Run:

srl bench skills/ evals.jsonl

Metrics:

• Top-1 accuracy
• Hit@K
• Mean Reciprocal Rank
• weighted Top-1 accuracy
• forbidden Top-1 rate
• ambiguity rate
• mean routing margin

5. Deterministic lexical baseline router

SkillRouteLab includes a local BM25-style baseline router.

It does not claim that lexical routing is the ideal production router.

It gives teams a deterministic baseline:

If a sophisticated LLM router cannot beat the local metadata-only baseline, the routing layer probably needs work.

The benchmark format is deliberately reusable so future versions can accept predictions from LLM or embedding routers.

6. Context-footprint analysis

For every skill:

• startup metadata token estimate
• activated SKILL.md token estimate
• body line count
• reference-file count
• asset-file count
• bundled-script count

Catalog totals include:

• startup discovery footprint
• average activated skill footprint
• referenced bytes

This connects Agent Skills to the broader context-engineering trend: a catalog should not only contain good workflows, it should expose them with a controlled context cost.

7. Progressive-disclosure audit

The Agent Skills specification recommends:

• compact discovery metadata
• loading full instructions only on activation
• moving conditional detail into reference files
• keeping the main skill body reasonably small

SkillRouteLab flags large monolithic skills that could make better use of references.

8. Cross-client portability profiles

Example profile:

{
  "name": "portable-agent-client",
  "supports_scripts": true,
  "supports_references": true,
  "allowed_tools_types": ["string"],
  "max_skill_body_tokens": 5000,
  "available_tools": ["Read", "Write", "Bash", "Grep"]
}

Then:

srl scan skills/ --profile client-profile.json

Checks include:

• scripts unsupported by client
• references unsupported by client
• allowed-tools representation mismatch
• declared tool unavailable
• skill activation body exceeding client budget

The profile system is descriptive rather than hard-coded to vendor behavior, because client implementations evolve quickly.

9. Community allowed-tools compatibility

The current open specification describes allowed-tools as a space-separated string.

In practice, community skills also use YAML lists.

SkillRouteLab:

• understands both for analysis
• reports list syntax as a portability note
• does not pretend community behavior overrides the specification

This distinction is important for real-world cross-client tooling.

10. Defensive static security review

SkillRouteLab never executes bundled scripts.

It statically flags high-impact patterns such as:

• remote script piped directly into a shell
• destructive rm -rf /
• chmod 777
• Python shell=True
• dynamic eval / exec
• os.system
• common credential patterns

These are review signals, not proof of malicious intent.

11. Stable catalog manifest

Generate:

srl manifest skills/ --output catalog.json

The manifest includes:

• names
• descriptions
• compatibility
• declared tools
• script/reference/assets
• context estimates
• SHA-256 skill-tree hash

Useful for:

• release review
• catalog inventory
• drift detection
• downstream tooling
• supply-chain workflows

12. SQLite regression history

Store catalog audit:

srl scan skills/ \
  --db skillroute.db \
  --name baseline

Store routing benchmark:

srl bench skills/ evals.jsonl \
  --db skillroute.db \
  --name routing-v1

List:

srl runs --db skillroute.db

Compare two runs of the same kind:

srl compare \
  --db skillroute.db \
  --baseline 1 \
  --candidate 2

Catalog deltas include:

• score
• errors
• warnings
• collision count
• startup tokens
• average activation tokens

Benchmark deltas include:

• Top-1
• Hit@K
• MRR
• forbidden-route rate
• ambiguity
• routing margin

13. CI quality gates

Catalog:

srl scan skills/ \
  --min-score 90 \
  --fail-on-error \
  --sarif skill-route.sarif

Routing:

srl bench skills/ evals.jsonl \
  --min-top1 0.90 \
  --max-ambiguity 0.20

14. Reports

Catalog audits:

• Console
• JSON
• Markdown
• HTML
• SARIF 2.1.0

Routing benchmark:

• Console
• JSON
• Markdown
• HTML

15. Mermaid skill graph

srl graph skills/ --output skills.mmd

The graph shows:

• skills
• referenced files
• scripts
• discovery collisions

Quick start

Install:

python -m pip install .

Audit the bundled example:

srl scan examples/skills \
  --profile examples/client-profile.json \
  --min-score 90 \
  --fail-on-error

Preview routing:

srl route examples/skills \
  "Review this pull request for bugs and missing tests"

Benchmark:

srl bench examples/skills examples/evals.jsonl \
  --top-k 3 \
  --min-top1 0.90

Generate outputs:

srl scan examples/skills \
  --json audit.json \
  --markdown audit.md \
  --html audit.html \
  --sarif audit.sarif

srl manifest examples/skills --output catalog.json
srl graph examples/skills --output skills.mmd

Benchmark dataset format

Minimum:

{
  "id": "case-1",
  "query": "Review this pull request",
  "expected_skill": "code-review"
}

Multiple acceptable skills:

{
  "id": "case-2",
  "query": "Check this patch before merge",
  "expected_skills": [
    "code-review",
    "secure-code-review"
  ]
}

Negative routing:

{
  "id": "case-3",
  "query": "Merge these PDFs",
  "expected_skill": "pdf-processing",
  "forbidden_skills": [
    "code-review"
  ]
}

Weighted cases:

{
  "id": "critical-1",
  "query": "Review this production auth patch",
  "expected_skill": "code-review",
  "weight": 3,
  "tags": [
    "critical",
    "security"
  ]
}

Rule families

SRL0xx  parsing / format
SRL1xx  specification / quality / context / portability
SRL2xx  defensive security signals
SRL3xx  client-profile compatibility
SRL4xx  discovery collisions

Project positioning

SkillRouteLab deliberately does not duplicate the official Agent Skills format validator.

The official validator answers:

Is this skill structurally valid?

SkillRouteLab asks:

Is this whole catalog discoverable, portable, context-efficient, regression-tested, and safe enough to distribute?

Why this maps to current GitHub trends

The open-source AI ecosystem in 2026 has moved strongly toward:

• reusable Agent Skills
• agent harness engineering
• persistent memory and context infrastructure
• context/token efficiency
• security scanning
• multi-agent orchestration

Skills are now a cross-runtime packaging layer used by projects and clients across the ecosystem.

The harder problem shifts from creating one skill to operating a large catalog reliably.

See SOURCES.md for the research sources used to scope v1.0.0.

Limitations

• lexical routing is a baseline, not semantic understanding
• static security checks can have false positives
• token estimates are approximate
• client profiles are user-supplied capability declarations
• SkillRouteLab does not execute scripts
• SkillRouteLab does not replace official Agent Skills validation

Roadmap

• import predictions from arbitrary LLM routers
• embedding-router adapter
• pairwise LLM routing judge
• under-trigger / over-trigger labeled eval mode
• catalog-scale confusion matrix
• HTML routing heatmap
• automatic description rewrite suggestions
• external dependency manifest proposal support
• MCP tool availability adapter
• Agent Skills registry integration
• GitHub repository URL scanner
• signed manifest support
• skill version drift analysis
• SBOM-like skill supply-chain manifest
• JUnit XML output
• GitHub PR annotations
• OpenTelemetry routing traces
• local interactive TUI
• Web dashboard
• large-catalog approximate collision search
• benchmark packs for public skill catalogs

License

MIT.