feat: matrix-parallelized multi-language CI/CD architecture with dynamic versioning by AR21SM · Pull Request #376 · numfocus/DISCOVER-Cookbook

AR21SM · 2025-08-19T15:42:32Z

Architecture Overview

This pull request implements a comprehensive refactoring of the documentation deployment infrastructure, transforming a monolithic single-threaded build pipeline into a distributed, matrix-parallelized CI/CD architecture. The implementation leverages advanced GitHub Actions orchestration patterns to achieve horizontal scaling across multiple language variants with smart version handling and safe, atomic deployments.

Core Technical Implementations

1. Matrix-Based Parallel Execution Framework

Before: Simple single-job workflow

jobs:
  deploy-book:
    steps:
      - name: Build the book
        run: bash ci/build_website.sh

After: Complex multi-job matrix system

jobs:
  get-info:
    outputs:
      languages: ${{ steps.get-languages.outputs.languages }}
      version: ${{ steps.version-info.outputs.version }}
  
  deploy-book:
    needs: get-info
    strategy:
      matrix:
        language: ${{ fromJSON(needs.get-info.outputs.languages) }}

This change enables parallel builds across all languages instead of sequential builds.

2. Dynamic Language Detection

Implemented a JSON-driven service discovery mechanism with runtime parsing capabilities:

- name: Get languages directly with jq
  id: get-languages
  run: |
    languages=$(jq -c '[.[] | .code]' DISCOVER/_static/languages.json)
    echo "languages=${languages}" >> "$GITHUB_OUTPUT"

Now adding a new language only requires updating the JSON file - no workflow changes needed

3. Automatic Version Detection

Built a Git context-aware version detection system:

- name: Determine version from GitHub context
  env:
    REF_NAME: ${{ github.ref_name }}
    REF_TYPE: ${{ github.ref_type }}
  run: |
    version="dev"
    if [[ "$REF_TYPE" == "tag" ]]; then
      version="${REF_NAME#v}"
    elif [[ "$REF_NAME" == *-translations ]]; then
      version="${REF_NAME%-translations}"
    fi
    echo "version=$version" >> "$GITHUB_OUTPUT"

The system automatically detects if you're on a tag, translation branch, or development branch and sets the version accordingly.

4. Advanced Branch Pattern Matching

on:
  push:
    branches:
      - main
      - '*-translations'
      - '!1.0-translations'  # Exclude 1.0-translations

This setup triggers builds for all translation branches but excludes specific legacy versions using negative patterns.

5. Environment Variable Injection Architecture

DISCOVER/conf.py modernization:

#  Old static configuration
version = 'dev'
language = 'en'

# New dynamic configuration
version = os.environ.get("WEBSITE_VERSION", "dev")
language = os.environ.get("WEBSITE_LANGUAGE", "en")

Sphinx configuration now operates through environment variable injection, enabling runtime parametrization of build contexts.

6. Parameterized Build Orchestration

ci/build_website.sh enhancement:

# Hierarchical parameter resolution with fallback chains
VERSION=${VERSION:-${1:-"dev"}}
LANGUAGE=${LANGUAGE:-${2:-"en"}}

echo "Building version: $VERSION, language: $LANGUAGE"

# Environment context propagation to Sphinx
WEBSITE_VERSION="$VERSION" WEBSITE_LANGUAGE="$LANGUAGE" sphinx-build -b html DISCOVER/ DISCOVER/_build/html

The build script now accepts parameters and passes them to Sphinx through environment variables.

7. Automatic Redirect Generation

- name: Create version redirect
  run: |
    mkdir -p temp_redirect
    cat > temp_redirect/index.html << 'EOF'
    <!DOCTYPE html>
    <html>
    <head>
        <meta charset="UTF-8">
        <title>DISCOVER Cookbook - Redirecting...</title>
        <meta http-equiv="refresh" content="0; url=./en/">
        <link rel="canonical" href="./en/">
    </head>
    <body>
        <a href="./en/">Redirecting to English...</a>
    </body>
    </html>
    EOF

Generates redirect infrastructure with canonical URL directives for optimized SEO and user navigation patterns.

8. Structured Deployment Architecture

- name: Deploy language to GitHub Pages
  with:
    publish_dir: ./DISCOVER/_build/html
    destination_dir: ${{ needs.get-info.outputs.version }}/${{ matrix.language }}
    keep_files: true

Each language gets deployed to its own directory: /version/language/ (like /dev/en/, /dev/es/, etc.)

9. Concurrency Management System

concurrency:
  group: "gh-pages"
  cancel-in-progress: true

Prevents multiple deployments from running at the same time and cancels old ones when new commits come in.

10. Root Deployment

Ensures root files are deployed only when running from the main branch.

deploy-root-files:
  needs: [get-info, deploy-book]
  runs-on: ubuntu-latest
  if: github.ref_name == 'main'  # Production branch only
  steps:
    - name: Create root files directory
      run: |
        mkdir -p root_files
        cp DISCOVER/404.html root_files/
        cp index.html root_files/
        cp DISCOVER/_static/versions.json root_files/

11. Directory Structure Schema

/
├── dev/
│   ├── en/          # English development artifacts
│   ├── es/          # Spanish development artifacts
│   └── index.html   # redirect
├── v2.0/
│   ├── en/          
│   ├── es/        
│   └── index.html   
├── index.html       # Landing Page
├── 404.html         #  error handling
└── versions.json    # Versions list

This system scales to handle dozens of languages without requiring workflow changes.

netlify · 2025-08-19T15:42:43Z

✅ Deploy Preview for stupendous-kringle-a86e81 ready!

Name	Link
🔨 Latest commit	`fc3b279`
🔍 Latest deploy log	https://app.netlify.com/projects/stupendous-kringle-a86e81/deploys/68aaabd93d137200080fd949
😎 Deploy Preview	https://deploy-preview-376--stupendous-kringle-a86e81.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

OriolAbril · 2025-08-19T17:14:30Z

This can't be merged until 2.0-translations and 1.0 translations have a ci/build_website.sh script and a _static folder with their respective languages

* update/workflow with parallel language builds and versioning * update/build_website.sh file * Remove/Tag deployment * add/concurrency * fix:all remaining changes * fix:environmment variable in conf.py * feat: deploy all languages including hidden ones --------- Co-authored-by: Oriol Abril-Pla <oriol.abril.pla@gmail.com>

AR21SM added 2 commits August 19, 2025 21:09

update/workflow with parallel language builds and versioning

50e9557

update/build_website.sh file

c472195

AR21SM marked this pull request as draft August 19, 2025 15:43

OriolAbril reviewed Aug 19, 2025

View reviewed changes

Comment thread .github/workflows/build-gh-pages.yml

Comment thread ci/build_website.sh Outdated

Comment thread .github/workflows/build-gh-pages.yml Outdated

Comment thread .github/workflows/build-gh-pages.yml Outdated

Comment thread ci/build_website.sh Outdated

AR21SM added 3 commits August 23, 2025 21:20

Remove/Tag deployment

7449a28

add/concurrency

09a8a68

fix:all remaining changes

fc3b279

OriolAbril changed the base branch from main to next August 24, 2025 06:21

OriolAbril and others added 2 commits August 24, 2025 08:26

Merge branch 'next' into update/build-gh-pages

b700527

fix:environmment variable in conf.py

ce22a76

AR21SM marked this pull request as ready for review August 24, 2025 07:03

OriolAbril reviewed Aug 26, 2025

View reviewed changes

Comment thread .github/workflows/build-gh-pages.yml

Comment thread ci/build_website.sh Outdated

feat: deploy all languages including hidden ones

d01f398

aterrel approved these changes Aug 26, 2025

View reviewed changes

OriolAbril approved these changes Aug 26, 2025

View reviewed changes

OriolAbril merged commit 86684d1 into numfocus:next Aug 26, 2025

This was referenced Aug 29, 2025

Update github deploy scripts #374

Closed

fix(deploy): ensure translations are built, docs: add link to translation guide #386

Closed

AR21SM changed the title ~~Update/build gh pages~~ feat: matrix-parallelized multi-language CI/CD architecture with dynamic versioning Aug 31, 2025

Navigation Menu

Uh oh!

Uh oh!

feat: matrix-parallelized multi-language CI/CD architecture with dynamic versioning - #376

feat: matrix-parallelized multi-language CI/CD architecture with dynamic versioning#376
OriolAbril merged 8 commits into
numfocus:nextfrom
AR21SM:update/build-gh-pages

AR21SM commented Aug 19, 2025 •

edited

Loading

Uh oh!

netlify Bot commented Aug 19, 2025 •

edited

Loading

Uh oh!

OriolAbril commented Aug 19, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Uh oh!

Conversation

AR21SM commented Aug 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Architecture Overview

Core Technical Implementations

1. Matrix-Based Parallel Execution Framework

2. Dynamic Language Detection

3. Automatic Version Detection

4. Advanced Branch Pattern Matching

5. Environment Variable Injection Architecture

6. Parameterized Build Orchestration

7. Automatic Redirect Generation

8. Structured Deployment Architecture

9. Concurrency Management System

10. Root Deployment

11. Directory Structure Schema

Uh oh!

netlify Bot commented Aug 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for stupendous-kringle-a86e81 ready!

Uh oh!

OriolAbril commented Aug 19, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

AR21SM commented Aug 19, 2025 •

edited

Loading

netlify Bot commented Aug 19, 2025 •

edited

Loading