feat: Add reproducible embeddings pipeline and interactive semantic + graph visualization

[marceloceccon]

Summary

This PR adds a fully reproducible way to generate semantic embeddings for the Marble Skill Taxonomy and an interactive visualization that lets you explore both semantic similarity and the prerequisite graph at the same time.

The original dataset intentionally excluded embeddings (see manifest.json, README.md, and CHANGELOG.md). This contribution provides a clean, Dockerized pipeline to compute them on demand while preserving the project's philosophy that embeddings are derived artifacts.

Motivation

The taxonomy is a rich graph of 1,590 micro-topics connected by 3,221 prerequisite edges. Being able to also view the concepts in semantic space reveals interesting structure:

• Conceptual clusters that cut across the hand-defined subjects/domains
• "Surprising" long-range prerequisite connections between semantically distant ideas
• The ability to visually dial between pure semantic layout and graph-respecting layout

What’s included

• embeddings/ — Dockerized pipeline (Dockerfile, generate.py, etc.)
  • Computes sentence embeddings (default: all-MiniLM-L6-v2)
  • Generates deterministic 2D + 3D UMAP layouts
  • Persists vectors to a local Chroma vector database
  • Writes a full provenance manifest (model, parameters, text strategy, timestamp)
• embeddings/viz.html — the source visualization template (now committed)
• scripts/generate-viz.sh — one-command entrypoint
• EMBEDDINGS.md — comprehensive documentation
• Updates to README.md, .gitignore, and package.json

The pipeline produces:

• generated/topics_with_layout.json (nodes + UMAP coordinates)
• generated/links.json
• generated/chroma/ (vector DB)
• generated/viz.html (copied from the source template)
• generated/generation_manifest.json

Usage

./scripts/generate-viz.sh

Then:

cd generated
python3 -m http.server 8080
# open http://localhost:8080/viz.html

Visualization highlights

• 3D force-directed graph seeded with UMAP positions
• Graph pull slider that morphs between pure semantic layout and prerequisite-respecting layout
• Click a node to highlight full prerequisite chains (with animated particles)
• "Highlight surprising prerequisites" – shows hard edges between distant concepts in embedding space
• Subject filters, search, age quick-filters, multiple color modes, camera controls, auto-rotate, and auto fit-view on load

Reproducibility

• Fully Dockerized
• Pinned dependencies
• Fixed random_state=42 for UMAP
• Explicit CPU inference
• Detailed manifest recording the exact model, UMAP params, and text construction strategy used

Notes

• embeddings/viz.html is now the committed source. The pipeline copies it into generated/ on each run.
• The visualization uses only open models and local storage. No external services or API keys are required.
• All generated artifacts remain in generated/ (correctly gitignored).

Testing

./scripts/generate-viz.sh
# verify that generated/viz.html, topics_with_layout.json, and chroma/ are created
# open the visualization and try the Graph pull slider + "Highlight surprising prerequisites"

---

Screenshots

See embeddings/screenshot.png (or run the pipeline locally for the live version).