docs: validate internal links

[nabinchha]

📋 Summary

Fixes the dead internal routes and fragments identified by the published-docs audit, then prevents regressions by validating links against Fern's navigation-derived URLs during the standard docs build. The validator understands nested sections, skip-slug, camel-case route splitting, Fern heading anchors, configured redirects, version aliases, MDX links, and notebook source content.

🔗 Related Issue

Fixes #826

🔄 Changes

✨ Added

• Add a deterministic internal route and fragment validator with actionable source locations and nearest-route suggestions.
• Add 13 validator tests covering Fern slug behavior, nested navigation, redirects, relative links, fragments, code blocks, and notebook-generated headings.
• Add make check-fern-links and run it automatically from make check-fern-docs.

🐛 Fixed

• Correct 39 broken internal link occurrences across current Fern pages.
• Replace stale plugin destinations with current authoring guides.
• Fix the seeding tutorial's stale repository-relative data link in both its source and generated Colab notebook.
• Update the moved StructEval repository URL and correct an invalid MCP provider fragment.

🔧 Changed

• Document the new required build check in the Fern maintainer guide and DataDesigner docs skill.
• Keep external URL checks outside required PR CI to avoid failures from bot protection, rate limiting, and transient network errors.

🔍 Attention Areas

⚠️ Reviewers: Please pay special attention to the following:

• fern/scripts/check-internal-links.py — route slugs and heading fragments intentionally use different Fern-compatible normalization rules.

🧪 Testing

• make check-fern-docs passes — internal links pass and Fern reports 0 errors
• make check-fern-links passes across 86 current authoring pages
• .venv/bin/pytest fern/scripts/tests -q — 25 passed
• Ruff lint and formatting checks pass for the new script and tests
• make check-license-headers passes
• E2E tests: N/A — docs build validation has focused unit and integration coverage

✅ Checklist

• Follows commit message conventions
• Commits are signed off (DCO)
• Maintainer documentation updated
• Architecture docs: N/A — no runtime architecture or public API changes

[github-actions]

Fern preview: https://nvidia-preview-pr-828.docs.buildwithfern.com/nemo/datadesigner

Fern previews include the docs-website version archive with PR changes synced into latest. Notebook tutorials are rendered without execution outputs in previews.

[github-actions]

Code Review: PR #828 — docs: validate internal links

Summary

This PR fixes dead internal doc links flagged by the #826 audit and, more importantly, adds a deterministic internal link/fragment validator (fern/scripts/check-internal-links.py) that is wired into make check-fern-docs so regressions are caught in CI. The validator derives canonical routes from Fern navigation titles (not file paths — the source of the old checker's false positives), and understands nested sections, skip-slug, camel-case splitting, heading anchors, configured redirects, version aliases, relative links, MDX href/id attributes, and notebook-generated headings.

Scope: 26 files, +570/-45. One new 333-line script, one 186-line test file (25 tests), 20 MDX/doc link corrections, Makefile wiring, and maintainer-doc updates. Structural risk is LOW — no runtime/public-API surface touched.

I verified locally:

• pytest fern/scripts/tests -q → 25 passed
• check-internal-links.py --root fern → "Checked internal links across 86 Fern pages." (matches PR claim)
• ruff check + ruff format --check on the new files → clean
• The seeding-notebook doc fix is accurate: the notebook does urllib.request.urlretrieve(...) a CSV (lines 118–122), so "The notebook downloads the source CSV before generation" correctly replaces the stale ../data repo-relative link.

Findings

Correctness — looks solid

• Route derivation (version_routes / build_index) correctly mirrors Fern: skip-slug sections are skipped, explicit slug overrides win, latest version gets both /x and /latest/x aliases, and / + /latest map to the first latest page. Well covered by test_build_index_uses_navigation_titles_and_skip_slug.
• Redirect matching (redirect_matches): re.escape then :param* → .* and :param → [^/]+, with the * variant substituted before the bare variant. Order is correct; verified against the real redirects block via test_..._redirects.
• Code masking (strip_code) preserves byte offsets and newlines so reported line numbers stay accurate — a nice touch, and the reason extract_links line math is trustworthy.
• Determinism: sources are walked in nav order, links are sorted(set(...)), and a seen set dedups (source, line, target). Output is stable across runs — appropriate for a required CI gate.

Minor / optional

1. Indented (4-space) code blocks are not masked. FENCE_PATTERN and HEADING_PATTERN both anchor with \s{0,3}, so a CommonMark indented code block (≥4 leading spaces, no fence) is not recognized as code. A [text](/does-not-exist) inside such a block would be treated as a live link → potential false positive. Fenced and inline code are handled; only the indented-code form is missed. Low likelihood in these docs, but worth a comment or a follow-up if it ever bites. (check-internal-links.py:82-99)

2. Duplicate-heading anchor suffixes assume Fern's scheme. extract_anchors disambiguates repeats as anchor, anchor-1, anchor-2 (:110-117). This matches common GitHub/Fern behavior but isn't verified against Fern's actual renderer for the 3+ collision case. Fine as an approximation; just flagging it as an assumption.

3. Notebook .py sources are scanned in addition to the generated .mdx. notebook_sources both (a) merges generated heading anchors into the wrapper page and (b) returns the .py files as additional link sources sharing the wrapper's route. This is intentional (it's how the 3-seeding .py broken link gets caught), but it means a link present in both the .py and its generated .mdx is validated twice under the same route. Harmless given the seen dedup is per-source-path, and arguably desirable defense-in-depth — noting only so it isn't mistaken for a bug.

4. check-fern-links depends on pyyaml via $(DOCS_PYTHON). The recipe runs the script with .venv/bin/python; yaml must be in the docs dependency group (it already is, used by fern-release-version.py). No action needed — just confirm the docs group, not a transitive extra, provides it so the required check can't break on a lean install.

Test coverage — strong

25 tests cover slugify vs anchor-slugify divergence (the flagged attention area), skip-slug nav, fragments, relative links, redirects, base-path prefixing, code-block/external-URL skipping, missing-route suggestions, missing fragments, and notebook heading extraction. The slugify parametrization (FileSystemSeedReader → file-system-seed-reader) directly locks in the camel-case behavior that produced several of the link corrections in this PR. Good regression protection.

Conventions

• SPDX headers present on both new files; from __future__ import annotations, absolute imports, full type annotations, modern str | None / list[str] syntax — all consistent with AGENTS.md / STYLEGUIDE.md.
• The script lives under fern/scripts/ (a docs-tooling location), not the data_designer namespace, so the interface→engine→config import rule doesn't apply. No layering concerns.
• Maintainer docs (fern/README.md, SKILL.md) and Makefile help/.PHONY are all updated in step — the old "known false positives" caveat in SKILL.md is correctly replaced.

Structural Impact

Risk: LOW (localized change) — 3 Python files, 0 AST entities, 0/82 clusters (graphify, 2.8s). No god nodes, no import-direction violations, no cross-package dependency changes.

Verdict

Approve (non-blocking). High-quality, well-tested, self-verifying change that replaces a false-positive-prone checker with a deterministic one and fixes real dead links. All local checks pass and the validator runs clean against the live docs. None of the findings block merge; items 1 and 4 are the most worth a moment's thought (indented-code false positives; confirming pyyaml is in the required docs dependency group). No correctness, security, or performance concerns.

(Per CI policy: this review does not approve or request changes on GitHub — it is posted as a comment by the workflow.)

[greptile-apps]

Greptile Summary

This PR adds a deterministic internal-link validator (check-internal-links.py) that builds a route index from Fern's navigation YAML and validates every link and fragment across the latest docs, then fixes 39 broken link occurrences that the new tool identified.

• New validator (fern/scripts/check-internal-links.py): builds a page route index from Fern navigation (handling skip-slug, camelCase splitting, version aliases, redirects, and relative links), strips fenced/inline code before scanning, and reports broken routes or fragments with nearest-route suggestions. Thirteen pytest tests cover all major code paths.
• Link fixes across 20+ MDX pages: the dominant correction is tool-use-and-mcp → tool-use-mcp (matching Fern's &-stripped slug), along with architecture-and-performance → architecture-performance, filesystemseedreader-plugins → file-system-seed-reader-plugins, a stale #mcpprovider-remote-sse fragment corrected to #mcpprovider-remote, stale plugin guide destinations replaced with current routes, and the seeding tutorial's repository-relative [data](../data) replaced with plain prose.
• Tooling integration: make check-fern-links is added and wired into make check-fern-docs so every standard docs build validates internal links going forward.

Confidence Score: 5/5

Safe to merge — all changes are docs and tooling; no runtime code is touched.

The validator logic is correct: route index construction from Fern navigation YAML, redirect pattern matching using re.escape + named-segment substitution, code-block masking with offset-preserving replacements, and anchor deduplication all behave as expected. Thirteen tests cover the relevant edge cases and the tool is confirmed passing across all 86 current authoring pages. The MDX link corrections are mechanically consistent (every broken slug is traced to the correct Fern-slugified navigation title), and the notebook source fix removes a repository-relative link that could never resolve in a Colab context.

No files require special attention.

Important Files Changed

Filename	Overview
fern/scripts/check-internal-links.py	New 351-line deterministic link validator: builds a route index from Fern navigation YAML, then checks every internal link and fragment across all pages. Logic for slugify/anchor-slugify, redirect pattern matching, code-block masking, and notebook markdown extraction all look correct.
fern/scripts/tests/test_check_internal_links.py	215-line pytest suite covering slug behaviour, nested navigation, skip-slug, redirect pattern matching, fragment validation, relative links, code-block exclusion, and notebook heading anchor injection. All parameterised cases match expected Fern URL conventions.
Makefile	Adds check-fern-links target and wires it into check-fern-docs; correctly gates on prepare-fern-docs and propagates failures through the sub-make call.
fern/versions/latest/pages/concepts/tool_use_and_mcp.mdx	Fixes 8 broken links: path segment changed from tool-use-and-mcp to tool-use-mcp across all sub-page references, and the stale #mcpprovider-remote-sse fragment corrected to #mcpprovider-remote to match the actual heading 'MCPProvider (Remote)'.
fern/versions/latest/pages/index.mdx	Two links to Architecture & Performance corrected from /concepts/architecture-and-performance to /concepts/architecture-performance, matching the Fern slug derived from the page title.
fern/versions/latest/pages/plugins/overview.mdx	Three links updated from the stale filesystemseedreader-plugins slug to file-system-seed-reader-plugins, and stale plugin guide destinations replaced with the active example-plugin and file-system-seed-reader-plugins routes.
fern/versions/latest/pages/devnotes/posts/structured-outputs-from-nemotron.mdx	Updates the StructEval repository URL from the stale StructEval/StructEval to the current TIGER-AI-Lab/StructEval GitHub path.
docs/notebook_source/3-seeding-with-a-dataset.py	Replaces a repository-relative relative link data (which resolved to a non-existent Fern route) with plain prose, fixing a broken link that was also propagated to the generated Colab notebook.
.agents/skills/datadesigner-docs/SKILL.md	Updates the docs skill to document that make check-fern-docs now validates internal links deterministically, replacing the previous note about known false positives in the old file-path-based checker.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[make check-fern-docs] --> B[prepare-fern-docs\ngenerate notebooks]
    A --> C[make check-fern-links]
    A --> D[fern check\nYAML + MDX validation]

    C --> E[build_index\nparse docs.yml + version YAMLs]
    E --> F[Walk navigation tree\nslugify section/page titles]
    F --> G[Compute routes\nlatest pages get /latest/ aliases]
    G --> H[Extract anchors\nheadings + explicit IDs per page]
    E --> I[Load redirects\nnormalize configured paths]

    C --> J[notebook_sources\nmap .py sources to wrapper page routes\nextract markdown-cell anchors]

    E --> K[validate_links\nfor each page source]
    J --> K
    K --> L{For each link}
    L --> M[normalize_target\nresolve relative / strip base path]
    M --> N{Path in index.routes?}
    N -- yes --> O{Fragment in page.anchors?}
    O -- pass --> P[valid]
    O -- fail --> Q[broken fragment]
    N -- no --> R{asset_exists or\nredirect_matches?}
    R -- yes --> P
    R -- no --> S[broken route\n+ closest_route suggestion]

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    A[make check-fern-docs] --> B[prepare-fern-docs\ngenerate notebooks]
    A --> C[make check-fern-links]
    A --> D[fern check\nYAML + MDX validation]

    C --> E[build_index\nparse docs.yml + version YAMLs]
    E --> F[Walk navigation tree\nslugify section/page titles]
    F --> G[Compute routes\nlatest pages get /latest/ aliases]
    G --> H[Extract anchors\nheadings + explicit IDs per page]
    E --> I[Load redirects\nnormalize configured paths]

    C --> J[notebook_sources\nmap .py sources to wrapper page routes\nextract markdown-cell anchors]

    E --> K[validate_links\nfor each page source]
    J --> K
    K --> L{For each link}
    L --> M[normalize_target\nresolve relative / strip base path]
    M --> N{Path in index.routes?}
    N -- yes --> O{Fragment in page.anchors?}
    O -- pass --> P[valid]
    O -- fail --> Q[broken fragment]
    N -- no --> R{asset_exists or\nredirect_matches?}
    R -- yes --> P
    R -- no --> S[broken route\n+ closest_route suggestion]

Loading

_{Reviews (4): Last reviewed commit: "Merge branch 'main' into codex/fix-docs-..." | Re-trigger Greptile}

[andreatnvidia]

LGTM. The navigation-aware link checker is well scoped, tested, and correctly wired into the docs build. Limiting validation to latest makes sense for immutable historical snapshots. Approved.