Huge revamp (#19)

* netlab-ui rounds 3–6: traffic, faults, captures, exercises, providers and showcase

- Multi-node shell, Ctrl+P context actions and smart run-on-nodes
- Traffic lens: live load, down links, errors/drops, tiles and sparklines
- Fault injection: link down/up and netem presets
- Packet capture with pcap downloads and live streaming
- Search by address, prefix, AS, VLAN, VRF or module
- Running-config snapshots, drift detection and diffs
- Lab time limits and per-user quotas; fork, publish and duplicate labs into workspaces
- Guided exercises with tasks, validation checks and saved progress
- libvirt and external providers; expanded netlab tools, setup, reports and custom configs
- Generated showcase gallery, feature previews and captioned tour video
- Animated loading intro and README logo outro with community credits
- Background music with seamless looping, automatic fades and attribution
- Keep video outputs and recording metadata local; commit gallery images and renderer sources
- Remove repository gallery links to ignored video files
- Move assistant MCP server/client to MCP 2
- Preserve netlab's short group form when saving

* Add CI jobs for hygiene checks and secrets scanning; update CONTRIBUTING.md for local commit checks

* Refactor README and GALLERY.md for improved layout; enhance error handling in router and config snapshot services; add tests for symlink validation

* Update badge colors in README for consistency and improved visibility

Author: Muddyblack

.github/workflows/ci.yml

@@ -12,6 +12,55 @@ concurrency:
   cancel-in-progress: true
 
 jobs:
+  hygiene:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - name: Install pre-commit
+        run: pip install pre-commit==4.3.0
+      - name: Check changed files
+        env:
+          SKIP: gitleaks
+          BASE_SHA: ${{ github.event.pull_request.base.sha }}
+          HEAD_SHA: ${{ github.event.pull_request.head.sha }}
+        run: |
+          set -euo pipefail
+          if [ "$GITHUB_EVENT_NAME" = "pull_request" ]; then
+            base=$(git merge-base "$BASE_SHA" "$HEAD_SHA")
+            pre-commit run --from-ref "$base" --to-ref "$HEAD_SHA" --show-diff-on-failure
+          else
+            pre-commit run --all-files --show-diff-on-failure
+          fi
+
+  secrets:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+      - uses: actions/setup-go@v5
+        with:
+          go-version: '1.25'
+          cache: false
+      - name: Install Gitleaks
+        run: go install github.com/zricethezav/gitleaks/v8@v8.30.1
+      - name: Scan every PR commit for secrets
+        env:
+          BASE_SHA: ${{ github.event.pull_request.base.sha }}
+          HEAD_SHA: ${{ github.event.pull_request.head.sha }}
+        run: |
+          set -euo pipefail
+          if [ "$GITHUB_EVENT_NAME" = "pull_request" ]; then
+            gitleaks git --redact --no-banner --log-opts="$BASE_SHA..$HEAD_SHA"
+          else
+            gitleaks git --redact --no-banner
+          fi
+
   # Decide which halves of the repo the PR actually touches, so a
   # frontend-only PR doesn't pay for a backend run (and vice versa).
   changes:
@@ -34,9 +83,11 @@ jobs:
 
           if [ "${{ github.event_name }}" != "pull_request" ]; then
             # Manual runs have no diff to inspect — exercise everything.
-            echo "backend=true" >> "$GITHUB_OUTPUT"
-            echo "frontend=true" >> "$GITHUB_OUTPUT"
-            echo "docker=true" >> "$GITHUB_OUTPUT"
+            {
+              echo "backend=true"
+              echo "frontend=true"
+              echo "docker=true"
+            } >> "$GITHUB_OUTPUT"
             exit 0
           fi
 
@@ -83,7 +134,7 @@ jobs:
           cache-dependency-path: backend/pyproject.toml
 
       - name: Install dependencies
-        run: pip install -e ".[dev,netlab,assistant]"
+        run: pip install -e ".[dev,netlab]"
 
       - name: Lint (ruff check)
         run: ruff check
@@ -127,6 +178,10 @@ jobs:
     needs: changes
     if: needs.changes.outputs.docker == 'true'
     runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        # Both published images: the UI-only default and the bundled toolchain.
+        target: [ui, full]
     steps:
       - uses: actions/checkout@v4
 
@@ -135,11 +190,12 @@ jobs:
       - uses: docker/build-push-action@v6
         with:
           context: .
+          target: ${{ matrix.target }}
           push: false
           # Single arch on PRs: this is a "does it build" gate, and emulating
           # arm64 here would roughly double the run for no extra signal.
           platforms: linux/amd64
-          # Shares the cache the release build populates, so tagging later is
-          # mostly cache hits.
-          cache-from: type=gha
-          cache-to: type=gha,mode=max
+          # Shares the per-target cache the release build uses (scope ui/full),
+          # so tagging later is mostly cache hits.
+          cache-from: type=gha,scope=${{ matrix.target }}
+          cache-to: type=gha,mode=max,scope=${{ matrix.target }}

.github/workflows/container-release.yml

@@ -40,22 +40,54 @@ jobs:
             type=semver,pattern={{major}}.{{minor}}
             type=semver,pattern={{major}}
 
+      # Same tags with a -full suffix (latest → latest-full) for the image that
+      # bundles netlab, Ansible and containerlab (Dockerfile target `full`).
+      - id: metadata-full
+        uses: docker/metadata-action@v5
+        with:
+          images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
+          flavor: |
+            suffix=-full,onlatest=true
+          tags: |
+            type=semver,pattern={{version}}
+            type=semver,pattern={{major}}.{{minor}}
+            type=semver,pattern={{major}}
+
       - uses: docker/setup-qemu-action@v3
       - uses: docker/setup-buildx-action@v3
 
       - uses: docker/build-push-action@v6
         with:
           context: .
+          target: ui
           # arm64 matters for Apple Silicon and arm-based lab hosts. Only the
           # Python stage is emulated — the frontend stage builds natively.
           platforms: linux/amd64,linux/arm64
+          build-args: |
+            NETLAB_GUI_VERSION=${{ github.ref_name }}
           push: true
           tags: ${{ steps.metadata.outputs.tags }}
           labels: ${{ steps.metadata.outputs.labels }}
           provenance: mode=max
           sbom: true
-          cache-from: type=gha
-          cache-to: type=gha,mode=max
+          # One cache scope per target, or the second build overwrites the first's.
+          cache-from: type=gha,scope=ui
+          cache-to: type=gha,mode=max,scope=ui
+
+      - uses: docker/build-push-action@v6
+        with:
+          context: .
+          target: full
+          platforms: linux/amd64,linux/arm64
+          build-args: |
+            NETLAB_GUI_VERSION=${{ github.ref_name }}
+          push: true
+          tags: ${{ steps.metadata-full.outputs.tags }}
+          labels: ${{ steps.metadata-full.outputs.labels }}
+          provenance: mode=max
+          sbom: true
+          cache-from: type=gha,scope=full
+          cache-to: type=gha,mode=max,scope=full
 
   release:
     needs: publish
@@ -65,6 +97,11 @@ jobs:
         with:
           fetch-depth: 0
 
+      # Image tags come from docker/metadata-action's semver patterns, which
+      # drop the leading "v" (tag v1.2.3 → image 1.2.3).
+      - id: version
+        run: echo "tag=${GITHUB_REF_NAME#v}" >> "$GITHUB_OUTPUT"
+
       # The GitHub Release is the landing page people reach from the tag, so
       # it leads with the pull command rather than only a source zip.
       - uses: softprops/action-gh-release@v2
@@ -76,27 +113,44 @@ jobs:
           body: |
             ## Run it
 
+            Everything included (netlab, Ansible, containerlab) — the host only
+            needs Docker. Download `docker-compose.yml` from this release, then:
+
             ```bash
+            docker compose up -d        # → http://localhost:8000
+            ```
+
+            or the equivalent `docker run`:
+
+            ```bash
+            mkdir -p "$PWD/labs"
             docker run -d --name netlab-ui \
-              -p 8000:8000 \
+              --privileged --network host --pid host \
               -v /var/run/docker.sock:/var/run/docker.sock \
-              -v "$PWD/labs:/work" \
-              ghcr.io/${{ github.repository }}:${{ github.ref_name }}
+              -v /var/run/netns:/var/run/netns \
+              -v "$PWD/labs:$PWD/labs" -e NETLAB_WORKSPACE="$PWD/labs" \
+              -v "$HOME/.netlab:/root/.netlab" \
+              -e UVICORN_HOST=127.0.0.1 \
+              ghcr.io/muddyblack/netlab-ui:${{ steps.version.outputs.tag }}-full
             ```
 
-            Then open <http://localhost:8000>.
+            The labs directory must be mounted at the **same path** as on the
+            host (containerlab bind-mounts node files through the host Docker
+            daemon), and `~/.netlab` keeps netlab's running-lab registry shared
+            with the host. The UI checks all of this itself: see
+            **Settings → Environment → Container Setup**.
 
-            Images: `linux/amd64`, `linux/arm64`. Also tagged `latest`.
+            ### UI-only image
 
-            ### Scope
+            `ghcr.io/muddyblack/netlab-ui:${{ steps.version.outputs.tag }}` (no `-full`)
+            leaves netlab, Ansible and containerlab on your host. Use the same
+            flags, plus mount your netlab install at its own path and point the
+            UI at it:
 
-            This image is the **UI only**. `netlab`, Ansible and containerlab
-            stay on your host — run it on a machine that already has netlab
-            set up, and point Settings → Environment at that install (or set
-            `NETLAB_BIN`).
+            ```bash
+              -v /opt/netlab-venv:/opt/netlab-venv:ro \
+              -e NETLAB_BIN=/opt/netlab-venv/bin/netlab \
+              -v /usr/bin/containerlab:/usr/bin/containerlab:ro
+            ```
 
-            Features that call containerlab directly — link impairment,
-            orphaned-instance cleanup, post-start link reconcile — need the
-            binary visible inside the container. It is a static Go binary, so
-            add `-v /usr/bin/containerlab:/usr/bin/containerlab:ro` to enable
-            them. Everything else works without it.
+            Images: `linux/amd64`, `linux/arm64`. Also tagged `latest` / `latest-full`.

.gitignore

@@ -1,5 +1,6 @@
 frontend/.npmrc
 .npmrc
+notes.md
 
 .vscode/
 

.pre-commit-config.yaml

@@ -0,0 +1,19 @@
+minimum_pre_commit_version: '3.2.0'
+repos:
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v6.0.0
+    hooks:
+      - id: trailing-whitespace
+        args: [--markdown-linebreak-ext=md]
+        # Preserve third-party patch context and generated SVG artwork.
+        exclude: '(^frontend/patches/|\.svg$)'
+      - id: check-merge-conflict
+        args: [--assume-in-merge]
+      - id: check-yaml
+        args: [--allow-multiple-documents]
+      - id: check-json
+      - id: detect-private-key
+  - repo: https://github.com/gitleaks/gitleaks
+    rev: v8.30.1
+    hooks:
+      - id: gitleaks

AGENTS.md

@@ -120,6 +120,16 @@ The netlab YAML stays coordinate-free. Positions live in `<topology>.netlab-ui.j
 - **Running-lab matching is by topology file, then the file's own directory** (`host/runningMatch.ts`). A directory *prefix* or bare name match marks unrelated labs as running. netlab allows one lab per directory — new labs are created as `<workspace>/<name>/topology.yml`.
 - **`host.createSession` does not make a session active.** Explorer actions on a lab that isn't the open tab get their own background session (`getOrCreateSession` in `useTabManager`); only `activateSession` changes what canvas callbacks target. Resolving an action through "the current session" ran Deploy/Destroy on the wrong lab before.
 - **The YAML serializer merges into the loaded document** (`serialize.to_yaml` + `Topology.source`) so comments and flow style survive UI edits. Rebuilding the document from the model dropped users' comments.
+- **`netlab exec` and `netlab connect --show` always exit 0.** `app/lab/broadcast.py` gets a real status for shell commands on containers from an exit-code trailer, and flags CLI error replies (`% Unknown command`) as failures. netlab joins the words after the node name and hands them to `bash -c`, so pass commands as whitespace-split words: `shlex` would strip the quotes, and an extra `sh -c` gets its `$?` expanded by the outer shell.
+- **clab-ui keeps only the traffic-rate keys of interface stats** (`EDGE_STATS_KEYS`). Anything else the runtime sample carries (error/drop counters, netem state) reaches our own UI through `host/runtimeStore.ts`, which the controller's runtime poll publishes to. Don't add a second poll.
+- **Dialogs and overlays opened from the canvas or explorer use small module stores** (`host/captureStore.ts`, `configsDialogStore.ts`, `copyLabStore.ts`, `canvasSpotlight.ts`), mounted once in `components/dialogs/LabToolDialogs.tsx` / `components/app/CanvasLabOverlays.tsx`. This avoids threading props through `App.tsx`; follow the same pattern for new ones.
+- **Explorer action ids choose their clab-ui menu group by substring** (`ACTION_GROUP_RULES`: `.graph.`, `copy`, `inspect`, `shell`, `addtoworkspace`, …). An id matching nothing lands in "Other".
+- **netlab's transformed topology stores `validate:` as a list** of tests with a `name` key, not the YAML's mapping. The same goes for other normalized blocks: read both forms.
+- **netlab's short group form (`core: [r1, r2]`) is a member list**, and `Group.short_form` keeps it that way on save.
+- **Providers are per node** (`node.provider`, else the lab's; netlab reports unmanaged devices as `unmanaged`). A lab with any non-clab node gets its canvas from `services/netlab/projection.py` (source `"transform"`), because `clab.yml` lists containers only. Gate runtime actions on the node's provider in the backend (`lifecycle._require_clab_node` / `_provider_limit`) with a message the user can act on, and keep `frontend/src/netlabProviders.ts` (the support table) and the README table in step.
+- **libvirt NICs map to netlab interfaces by order** (`services/netlab/libvirt.py`): `virsh domiflist` lists the management NIC first, then one NIC per non-virtual, non-loopback interface. p2p links are UDP tunnels (no host tap, target `-`); only LAN links have a `vnetN` tap. Tap counters are the host's view, so rx/tx are swapped. No KVM in CI: test with the fake `virsh` in `tests/test_libvirt.py`.
+- **netlab's Ansible playbooks find the lab through `$PWD`**, not the process cwd (`lookup('env','PWD')` in collect-configs, initial-config: node_files). `runner._child_env(cwd)` sets `PWD`. Any new way of spawning netlab must do the same, or configs land in the backend's own directory.
+- **Things netlab only knows from the deployed snapshot run in netlab's own Python** (`services/netlab/_tools_bridge.py`, the catalog script in `services/netlab/setup.py`, run with `location.target_python()`), never imported into the backend. `netlab test` exits 0 even when the test failed, and waits for RETURN before cleaning up: `setup.stream` answers the prompt, checks the output for failure, and tears the test lab down in a shielded task.
 
 ## Key rules
 
@@ -129,12 +139,16 @@ Never put netlab text, branding, or logic into clab-ui — it's a third-party de
 
 ### Installing / upgrading clab-ui
 
-`@containerlab/clab-ui` is published on the public npm registry (source: the [srl-labs/containerlab-app](https://github.com/srl-labs/containerlab-app) monorepo), pinned in `frontend/package.json`. No auth is needed to install. To upgrade, bump the version, port the sequenced patches in `frontend/patches/` (`@containerlab+clab-ui+<version>+NNN+<name>.patch`: 001 initial fixes, 002 lifecycle-modal lab name + actions, 003 node-editor `hostFields`), and run `npm install` — there's no build step on our side. Keep each patch generic (no netlab text) and small; add new ones with `npx patch-package @containerlab/clab-ui --append <name>`. After changing a patch, restart Vite with `--force` (its dependency pre-bundle cache keeps the old code).
+`@containerlab/clab-ui` is published on the public npm registry (source: the [srl-labs/containerlab-app](https://github.com/srl-labs/containerlab-app) monorepo), pinned in `frontend/package.json`. No auth is needed to install. To upgrade, bump the version, port the sequenced patches in `frontend/patches/` (`@containerlab+clab-ui+<version>+NNN+<name>.patch`: 001 initial fixes, 002 lifecycle-modal lab name + actions, 003 node-editor `hostFields`, 004 link down/up menu item), and run `npm install` — there's no build step on our side. Keep each patch generic (no netlab text) and small; add new ones with `npx patch-package @containerlab/clab-ui --append <name>`. After changing a patch, restart Vite with `--force` (its dependency pre-bundle cache keeps the old code).
 
 ### Public API only
 
 Import from `@containerlab/clab-ui` (the package root or its declared sub-entries like `@containerlab/clab-ui/host`). Never use `@containerlab/clab-ui/src/...` or `@containerlab/clab-ui/dist/...` deep paths.
 
+### Screenshots and videos are generated
+
+`docs/showcase/` drives the real app with Playwright and produces the README gallery, per-feature stills (light and dark), clips and one shareable video (`docs/showcase/run.sh`). A new user-facing feature gets a file in `docs/showcase/features/` (see its README); after UI changes, rerun `run.sh` (or `--only <feature>`) rather than hand-editing images. Commit the gallery screenshots and animated WebP previews. MP4s, intro/outro renders, recording manifests, caption timings, `.work/` and `.cache/` stay local and are ignored. Publish videos with their manifest and timings as release assets or external artifacts; do not add repository links to ignored videos.
+
 ### API types are generated
 
 `frontend/src/api/generated.ts` is auto-generated from the backend's OpenAPI schema. Never hand-edit it. After changing a Pydantic response model, run `npm run gen:api` (backend must be running) then `npm run typecheck`.