Skip to content

README.md

Latest commit

 

History

History
148 lines (110 loc) · 8.62 KB

README.md

File metadata and controls

148 lines (110 loc) · 8.62 KB
Raw

Crawl and publish weekly data Deploy Hugo site

Claracle (SquadScope)

Claracle is an automated AI-powered site that publishes weekly, monthly, and yearly summaries of tech trends sourced from GitHub. The system crawls trending and newly-created repositories, applies AI analysis to identify signal vs. noise, and publishes curated insights with zero manual intervention.

What is Claracle?

Claracle solves the information overload problem in open-source development. Each week, millions of repositories are created or gain stars. Claracle:

  1. Crawls GitHub API for new and trending repositories in a given week
  2. Analyzes the data with Copilot to identify what's genuinely important vs. hype
  3. Generates human-readable weekly summaries with signal, noise, and gaps analysis
  4. Publishes to GitHub Pages with RSS feeds for consumption
  5. Reskills every 5 runs to improve its own analysis quality

Result: Curated tech trend insights delivered automatically every week.

Architecture

Claracle uses a 5-stage pipeline executed entirely in GitHub Actions:

Crawl → Analyze → Generate → Deploy → Reskill
  ↓        ↓          ↓         ↓        ↓
JSON    Markdown   Hugo      Pages    Improvements

Stage 1: Crawl (scripts/crawl.py, scripts/techcrunch_crawler.py)

  • Queries GitHub API for repos created/trending in the current week
  • Fetches configured external RSS feeds from config/external_news_sources.json in parallel as an enrichment signal
  • Applies heuristic filtering (language, topic, description quality)
  • Outputs: data/raw/YYYY-WNN.json, data/raw/YYYY-WNN-external-news.json, data/snapshots/YYYY-WNN-stars.json

Stage 2: Analyze (Copilot CLI agents, gpt-5.5)

  • Two-step AI analysis using dedicated Copilot CLI agents configured in .github/agents/:
    1. Weekly Synthesis (weekly-synthesis) — generates a compact industry narrative (~2 000 tokens) from press context and historical signals
    2. Weekly Analysis (weekly-analysis) — consumes the synthesis plus raw crawl data to produce the full editorial with signal/noise/gaps classification
  • Both agents use gpt-5.5 (configured in agent frontmatter) with only read/write tools
  • Outputs: data/analyzed/YYYY-WNN-summary.md with quality score, data/analyzed/YYYY-WNN-correlations.json, data/analyzed/YYYY-WNN-press-context.md
  • Quality gate: Blocks publish if quality_score < 60 or missing required sections

Stage 3: Generate (scripts/generate_content.py)

  • Converts analyzed Markdown into Hugo content structure
  • Outputs: content/weekly/YYYY/WNN.md ready for Hugo build

Stage 4: Deploy (Hugo + GitHub Pages)

  • Builds static site from Hugo and publishes to GitHub Pages
  • Outputs: Live website + RSS feed at https://www.claracle.com/index.xml

Stage 5: Reskill (every 5th run)

  • Every 5th run, Copilot reviews squad history and recent analysis outputs
  • Writes observations and improvement recommendations to .squad/reskill/YYYY-WNN.md
  • Optional: Can trigger PR with proposed prompt refinements (not auto-merged)

Theme and stack

  • Static site generator: Hugo (extended, v0.146.0+)
  • Theme: PaperMod
  • Search: Pagefind (static, client-side)
  • Notifications: RSS feeds + GitHub Releases
  • Automation: GitHub Actions
  • Deployment: GitHub Pages
  • Analysis engine: Copilot CLI agents (weekly-synthesis, weekly-analysis) using gpt-5.5; no GitHub Models/OpenAI fallback

Quick start

For end users

  1. Visit the site: Claracle: Browse weekly, monthly, yearly summaries
  2. Subscribe to RSS: Add https://www.claracle.com/index.xml to your reader
  3. Check GitHub Releases: New summaries also posted as releases

For operators (see docs/operator-guide.md for full setup)

  1. Fork this repository
  2. Configure COPILOT_GH_TOKEN secret in your repo (fine-grained PAT with Copilot Requests permission)
  3. Enable GitHub Pages (Actions source)
  4. Test manual run: gh workflow run crawl-and-publish.yml
  5. Monitor the first automated run

⚠️ First-time operators: Start with docs/rollout-checklist.md to ensure all prerequisites are in place.

Local development

  1. Install Hugo: brew install hugo (macOS) or download from hugo releases (v0.146.0 or newer)
  2. Clone with submodules: 
    git clone --recurse-submodules https://github.com/jmservera/SquadScope.git
git submodule update --init --recursive
  3. Start dev server: hugo server (default: http://localhost:1313)
  4. Create production build: hugo --minify (output: public/)

Content structure

  • content/weekly/YYYY/WNN.md — immutable weekly summaries (published once, never modified)
  • content/monthly/YYYY/MM.md — monthly rollups with synthesis narrative, SEO editorial titles (max 70 chars), cross-links to weekly/yearly pages, trend-arc sections, and structured frontmatter (themes, gaps, repos)
  • content/yearly/YYYY.md — yearly narrative summaries with SEO titles, ≤155-char meta descriptions, and cross-links to monthly pages
  • data/raw/YYYY-WNN.json — GitHub crawler output (JSON object with keys: week, new_repos, trending_repos, signals, metadata)
  • data/raw/YYYY-WNN-external-news.json — external RSS enrichment output from sources configured in config/external_news_sources.json
  • data/analyzed/YYYY-WNN-summary.md — AI analysis with quality score
  • data/snapshots/YYYY-WNN-stars.json — star count snapshots for trending analysis

Automated weekly pipeline

.github/workflows/crawl-and-publish.yml runs the full weekly automation on a best-effort Sunday schedule at 11:53 UTC:

  1. Crawl: GitHub API → data/raw/YYYY-WNN.json; external RSS feeds → data/raw/YYYY-WNN-external-news.json
  2. Analyze: Copilot → data/analyzed/YYYY-WNN-summary.md
  3. Quality gate: Validates quality_score ≥ 60; blocks publish if failed
  4. Generate: Markdown → content/weekly/YYYY/WNN.md
  5. Deploy: Hugo build → GitHub Pages
  6. Reskill: Every 5th run, review and improve squad state

Schedule and manual runs

  • Automated schedule: Sunday 11:53 UTC (53 11 * * 0) on GitHub-hosted runners
  • Timing expectation: GitHub Actions schedule is best-effort on shared runners; this repo has observed multi-hour delays on scheduled starts
  • Manual trigger: gh workflow run crawl-and-publish.yml or GitHub Actions UI
  • If punctuality matters: trigger the existing workflow_dispatch from an external scheduler; see docs/operator-guide.md#schedule-latency-and-mitigation-ladder

Required secrets

  • COPILOT_GH_TOKEN — Fine-grained PAT with Account → Copilot Requests permission for Copilot CLI analysis
  • GITHUB_TOKEN — Built-in; used for crawling, commits, Pages deployment, and issue/notification automation
  • PODCASTER_API_KEY — Optional; used with the PODCASTER_ENDPOINT Actions variable for the post-publish Podcaster handoff. The value must never be logged or committed.

Crawler notes

  • signals.top_topics de-duplicates repositories by full_name across new and trending buckets
  • data/snapshots/YYYY-WNN-stars.json preserves the broader pre-filter candidate set for consistent week-over-week stars_gained comparisons
  • Live runs use open-ended created:> / pushed:> GitHub search filters
  • --as-of mode switches to bounded date ranges for deterministic historical backfills

Deployment

  • Standard deploys: Direct pushes to main trigger .github/workflows/deploy-site.yml
  • Weekly automation: crawl-and-publish.yml handles full pipeline and deploys Pages
  • Deduplication: Deploy workflow skips bot-authored pushes to avoid duplicate Pages runs

Documentation

  • docs/operator-guide.md — Complete setup, configuration, and troubleshooting guide for new operators
  • docs/rollout-checklist.md — Step-by-step verification checklist for first-time deployments
  • docs/pipeline-validation.md — Stage checklist, artifact handoffs, success criteria, and known limitations
  • docs/analysis-spec.md — Detailed specification for analysis output format and quality gate criteria
  • .squad/decisions.md — Architectural decisions and decision history