Skip to content

architecture.mdx

Latest commit

 

History

History
378 lines (274 loc) · 9.68 KB

architecture.mdx

File metadata and controls

378 lines (274 loc) · 9.68 KB
Raw
title Architecture
description System architecture and data flow

Architecture

System Overview

The Leaderboard system follows a build-time data aggregation pattern with static site generation.

graph TB
    GitHubAPI[GitHub API]
    SlackAPI[Slack API]
    OtherSources[Other Sources]

    GitHubPlugin[GitHub Plugin]
    SlackPlugin[Slack Plugin]
    CustomPlugin[Custom Plugin]

    PluginRunner[Plugin Runner]
    LibSQL[(LibSQL Database)]

    ContributorsMD[contributors/*.md]
    ActivitiesJSONL[activities/*.jsonl]

    NextBuild[Next.js Build]
    StaticSite[Static Site]

    GitHubAPI --> GitHubPlugin
    SlackAPI --> SlackPlugin
    OtherSources --> CustomPlugin

    GitHubPlugin --> PluginRunner
    SlackPlugin --> PluginRunner
    CustomPlugin --> PluginRunner

    ContributorsMD --> PluginRunner
    ActivitiesJSONL --> PluginRunner

    PluginRunner --> LibSQL
    PluginRunner --> ContributorsMD
    PluginRunner --> ActivitiesJSONL

    LibSQL --> NextBuild
    NextBuild --> StaticSite
Loading

Components

1. Plugin Runner (@leaderboard/plugin-runner)

Purpose: Orchestrate data collection and management

Responsibilities:

  • Load configuration from config.yaml
  • Fetch and validate plugins from URLs
  • Import existing data from data-repo
  • Execute plugin setup, scrape, and aggregate methods
  • Evaluate config and plugin badge rules
  • Export updated data back to data-repo

CLI Commands:

plugin-runner                          # Run all phases
plugin-runner import                   # Import data only
plugin-runner setup                    # Plugin setup only
plugin-runner scrape                   # Plugin scrape only
plugin-runner aggregate                # Aggregation only
plugin-runner evaluate                 # Badge evaluation only
plugin-runner export                   # Export data only
plugin-runner --data-dir=/path/to/data # Custom data directory

2. API Layer (@ohcnetwork/leaderboard-api)

Purpose: Unified database utilities and plugin type definitions

Technology: LibSQL (SQLite-compatible)

Schema:

  • contributor - User profiles and metadata
  • activity_definition - Activity types defined by plugins
  • activity - Individual activity records
  • global_aggregate - Organization-level metrics
  • contributor_aggregate - Per-contributor metrics
  • badge_definition - Badge types and variants
  • contributor_badge - Badges earned by contributors

Storage:

  • Default location: ${DATA_DIR}/.leaderboard.db
  • Persists activity_definition table across runs
  • Temporary storage for contributor and activity tables

Provides:

  • Database abstraction layer
  • Query builders and helpers
  • TypeScript type definitions
  • Plugin interface specification
  • Context object structure

Example Plugin:

import type { Plugin } from "@ohcnetwork/leaderboard-api";

export default {
  name: "my-plugin",
  version: "1.0.0",
  badgeDefinitions: [
    // Optional: Define custom badges
  ],
  badgeRules: [
    // Optional: Define custom badge evaluation rules
  ],
  async setup(ctx) {
    // Define activity types
  },
  async scrape(ctx) {
    // Fetch and store activities
  },
  async aggregate(ctx) {
    // Optional: Compute plugin-specific aggregates
  },
} satisfies Plugin;

3. MCP Server (@leaderboard/mcp-server)

Purpose: Enable AI assistants to query leaderboard data via Model Context Protocol

Features:

  • 20+ query tools for comprehensive data access
  • Dual transport support (STDIO and HTTP)
  • Read-only operations for security
  • Batch query operations
  • Input validation with Zod schemas

Tools Categories:

  • Contributor Tools: Query contributors, get stats, batch operations
  • Activity Tools: Query activities, search, timeline analysis
  • Leaderboard Tools: Rankings, top contributors, active users
  • Badge Tools: Badge definitions, awards, top earners
  • Aggregate Tools: Global and contributor-level metrics

Usage:

# STDIO transport (for Claude Desktop)
leaderboard-mcp --data-dir ./data

# HTTP transport
leaderboard-mcp --transport http --port 3001

Integration:

  • Claude Desktop configuration
  • MCP-compatible AI clients
  • Natural language queries

See MCP Server Documentation for detailed usage.

4. Next.js Application (apps/leaderboard-web)

Purpose: Generate static website

Features:

  • Server-Side Generation (SSG) at build time
  • Reads data directly from LibSQL database
  • Markdown documentation via Fumadocs
  • Customizable themes via CSS

Pages:

  • / - Home page with overview
  • /leaderboard - Rankings and leaderboards
  • /people - All contributors
  • /[username] - Individual contributor profiles
  • /badges - Badge definitions and achievements
  • /data - Data Explorer (browser-based SQL REPL)
  • /docs - Documentation

Data Flow

Build Process

  1. Import Phase

    data-repo/contributors/*.md → LibSQL
data-repo/activities/*.jsonl → LibSQL

  2. Setup Phase

    For each plugin:
  Execute plugin.setup(ctx)
  → Populate activity_definition table

  3. Scrape Phase

    For each plugin:
  Execute plugin.scrape(ctx)
  → Fetch data from API
  → Insert activities into database

  4. Aggregation Phase

    Calculate standard global aggregates
Calculate standard contributor aggregates

  5. Plugin Aggregation Phase

    For each plugin (with aggregate method):
  Execute plugin.aggregate(ctx)
  → Compute plugin-specific aggregates

  6. Badge Evaluation Phase

    Evaluate badge rules from config (threshold, streak, growth, composite)
For each plugin (with badgeRules):
  Evaluate plugin badge rules

  7. Export Phase

    LibSQL → data-repo/contributors/*.md
LibSQL → data-repo/activities/*.jsonl

  8. Build Phase

    Next.js pre-build setup:
  → Copy theme overrides
  → Copy database to public/data.db
  → Copy custom assets from data-repo
  → Download & optimize contributor avatars (WebP, 256×256)
Next.js reads LibSQL
→ Generate static pages
→ Output static site

Avatar Optimization

Contributor profiles store avatar_url pointing to external sources (e.g., GitHub avatars). During the build phase, these are downloaded, converted to WebP, and resized to 256×256 using sharp. The optimized images are saved to public/avatars/{username}.webp and served locally, eliminating external dependencies at page load.

Avatars are cached between builds — only new or missing avatars are downloaded, making subsequent builds fast.

Client-Side Data Explorer

In addition to the pre-rendered static pages, the database file is shipped as a static asset (public/data.db). The Data Explorer page loads this database in the browser via sql.js-httpvfs (SQLite compiled to WebAssembly). A Web Worker makes HTTP range requests to fetch only the database pages needed for each query, enabling ad-hoc SQL queries without any backend.

graph LR
    CDN["Static CDN\n(data.db)"] -->|"HTTP Range Requests"| Worker["Web Worker\n(sql.js WASM)"]
    Worker -->|"Query Results"| UI["Data Explorer UI"]
    UI -->|"SQL Query"| Worker
Loading

Data Storage Strategies

Contributors (Markdown Files)

Format: Markdown with YAML frontmatter

Rationale:

  • Human-editable profiles
  • Supports rich bio content
  • Version control friendly

Location: data-repo/contributors/<username>.md

Activities (Sharded JSONL)

Format: JSON Lines, one file per contributor

Rationale:

  • Efficient for large datasets
  • Easy per-user updates
  • Fast import/export

Location: data-repo/activities/<username>.jsonl

Activity Definitions (Database Only)

Format: SQLite table

Rationale:

  • Managed by plugins
  • No manual editing needed
  • Avoids sync issues

Location: data-repo/.leaderboard.db

Deployment Architecture

graph LR
    DataRepo[Data Repository]
    CI[CI/CD Pipeline]
    BuildServer[Build Server]
    CDN[Static CDN]
    Users[Users]

    DataRepo -->|Clone| CI
    CI -->|Run Plugin Runner| BuildServer
    BuildServer -->|Next.js Build| BuildServer
    BuildServer -->|Deploy| CDN
    CDN -->|Serve| Users
Loading

Steps:

  1. CI/CD clones data repository
  2. Runs plugin-runner to update data
  3. Builds Next.js static site
  4. Deploys to CDN (Netlify, Vercel, etc.)
  5. Users access static site from CDN

12-Factor App Compliance

The system follows 12-Factor App principles:

  1. Codebase: Single repo, multiple deployments
  2. Dependencies: Explicitly declared in package.json
  3. Config: Environment variables and config.yaml
  4. Backing Services: LibSQL as attachable resource
  5. Build/Run/Release: Clear separation of phases
  6. Processes: Stateless static site
  7. Port Binding: Not applicable (static export)
  8. Concurrency: Plugin execution parallelizable
  9. Disposability: Fast startup, clean shutdown
  10. Dev/Prod Parity: Same build process everywhere
  11. Logs: Structured logging in plugin-runner
  12. Admin Processes: Plugin-runner as separate process

Security Considerations

Plugin Execution

  • Plugins fetched from configurable URLs
  • Basic validation of plugin structure
  • Consider using signed plugins in production

Data Privacy

  • All data stored in your infrastructure
  • No external data transmission (except plugin API calls)
  • Contributor data fully under your control

Static Site

  • No server-side code execution
  • No authentication required
  • Can be deployed behind authentication if needed