awesome-hub-generator

English | 简体中文

awesome-hub-generator turns a research direction into a living awesome hub: papers, datasets, GitHub projects, citation signals, bilingual summaries, deep-reading notes, trend pages, and GitHub Pages deployment.

It is built for research areas that move too fast for a hand-maintained list. Give it a topic, seed sources, and model keys; it will discover papers from arXiv, Hugging Face, upstream awesome lists, GitHub projects, OpenAlex, and configured seed graphs, then publish a browsable static website plus machine-readable metadata.

中文说明请阅读 README.zh-CN.md。

Generated Hubs

These are real hubs generated and maintained with this repository. Each one has both a browsable website and a normal GitHub repository with structured metadata.

Hub	Website	GitHub	Focus	What it demonstrates
Awesome World Model Hub	中文 / English	jin-s13/awesome-world-model-hub	World models, interactive video models, embodied AI, simulation, benchmarks	Full research hub with papers, datasets, projects, citation ranking, teaser recovery, literature analysis, and daily GitHub Actions updates
Awesome AI4CAD Hub	中文 / English	jin-s13/awesome-AI4CAD-hub	AI for CAD, Neural CAD, B-Rep modeling, text-to-CAD, CAD reconstruction	Domain-specific discovery from arXiv, upstream awesome lists, GitHub project mining, and AI-for-CAD taxonomy
Awesome Auto Research Hub	English / 中文	jin-s13/awesome-auto-research-hub	AI research agents, automated science, paper writing, experiment automation	Literature-review synthesis, autonomous research-system tracking, datasets, projects, and deployable daily-update workflow

These hubs are normal downstream repositories, not submodules. The generator owns the tooling and templates; each hub owns its data, assets, website, and deployment history.

What It Builds

Surface	Output
Paper index	Searchable cards with scores, TLDRs, tags, links, and teaser images
Paper detail pages	Score breakdown, reasoning, keyword relevance, deep analysis, and related resources
Research analysis	Topic-level synthesis, shared patterns, differences, trend evolution, and representative papers
Dataset index	Dataset and benchmark entries discovered from papers, Hugging Face, and upstream awesome sources
Project index	Relevant GitHub projects with stars, tags, source repos, and paper links when available
Trends	Keyword trends, year distributions, score patterns, and taxonomy slices
Metadata	Structured data/, assets/, and resource/ files for downstream reuse
Deployment	GitHub Actions workflow template for daily refresh and GitHub Pages deployment

Why This Exists

Traditional awesome lists are beautiful but static. Research discovery needs a slightly different shape:

• Read-first ranking: prioritize papers by topical fit, recency, citation signal, reproducibility, methodology evidence, and configured research intent.
• LLM-assisted summaries: generate TLDRs for every paper and deeper analysis for stronger candidates.
• Source fusion: combine arXiv, upstream awesome repos, Hugging Face datasets, OpenAlex, Semantic Scholar expansion, GitHub project discovery, and manual overrides.
• Bilingual sites: publish English and Chinese reading surfaces from the same metadata.
• Daily updates: keep the hub alive without turning every refresh into a manual curation session.

Quick Start

1. Create a local hub

uv run python scripts/init_hub.py --name awesome-cad-hub --title "Awesome CAD Hub"

This creates .local/awesome-cad-hub/ with:

awesome.yaml
data/
assets/
resource/
website/

2. Configure the research direction

Edit .local/awesome-cad-hub/awesome.yaml:

project:
  name: "Awesome CAD Hub"
  description: "A curated hub for CAD papers, datasets, projects, and Neural CAD research."
  site_url: "https://your-username.github.io/awesome-cad-hub"

research:
  keywords:
    - "CAD"
    - "B-Rep"
    - "parametric CAD"
    - "text-to-CAD"

  negative_keywords:
    - "medical imaging"
    - "weather forecast"

  arxiv_categories:
    - "cs.CV"
    - "cs.GR"
    - "cs.LG"

  date_from: "2020-01-01"
  daily_search_days: 3

3. Configure model and data keys

For local runs, create .env in the hub or generator workspace:

cp .env.example .env

Common keys:

Name	Required	Purpose
ARK_API_KEY	Yes for LLM runs	Scoring, TLDRs, translation, deep analysis
ARK_API_BASE_URL	Optional	OpenAI-compatible /responses endpoint
ARK_MODEL_NAME	Optional	Fast scoring model
SMART_MODEL_NAME	Optional	Deep analysis and synthesis model
ARK_SURVEY_MAX_TOKENS	Optional	Max output tokens for bilingual literature-review synthesis
MINERU_API_KEY	Optional	PDF parsing for teaser extraction, figure/table extraction, and metadata fallback
OPENALEX_API_KEY	Optional	Citation metadata
OPENALEX_MAILTO	Optional	OpenAlex polite pool email
SEMANTIC_SCHOLAR_API_KEY	Optional	Reference expansion
GH_TOKEN	Optional local	GitHub discovery rate limits
GH_DISCOVERY_TOKEN	Optional Actions	GitHub discovery token for workflows

System tools used by optional enrichment paths:

Tool	Required	Install	Purpose
pdftotext	Optional	macOS: brew install poppler; Ubuntu/Debian: apt-get install poppler-utils	Local PDF fallback for author and affiliation extraction when arXiv HTML/abs/TeX metadata is incomplete and MinerU is unavailable

If MinerU is configured, metadata enrichment uses MinerU before pdftotext. If both MinerU and pdftotext are unavailable, builds continue with HTML/abs/TeX metadata extraction and skip only the PDF fallback path.

4. Build the first version

uv run python scripts/build.py --hub awesome-cad-hub

The first build discovers papers, filters relevance, scores papers, fetches teasers, generates interpretations, builds analysis pages, and renders an Astro website. Existing metadata and LLM cache entries are reused on reruns.

5. Preview locally

uv run python scripts/serve_hub.py --hub awesome-cad-hub --port 4327

serve_hub.py pulls the hub checkout first when it is clean, then starts the Astro dev server. Dirty checkouts are not pulled, so local edits are not overwritten.

Managing Multiple Local Hubs

Generated hubs live under .local/ as ordinary git checkouts. They are managed through hubs.yaml and scripts/hubctl.py.

uv run python scripts/hubctl.py list
uv run python scripts/hubctl.py status
uv run python scripts/hubctl.py pull
uv run python scripts/hubctl.py push awesome-world-model-hub
uv run python scripts/hubctl.py serve awesome-world-model-hub --port 4327
uv run python scripts/hubctl.py update awesome-world-model-hub --search-days 14 --skip-build

The default hubs.yaml registers:

• .local/awesome-world-model-hub
• .local/awesome-ai4cad-hub

This keeps hub repositories easy to operate from the generator workspace without using git submodules or submodule pointer commits.

Deploy a Hub Repository

When you want a public website, create a standalone downstream repository and let the generator prepare the deployable file set.

uv run python scripts/init_site.py --name awesome-cad-hub --title "Awesome CAD Hub"

For an existing local hub under .local/, use the deployment helper:

uv run python scripts/deploy_hub.py \
  .local/awesome-cad-hub \
  --repo jin-s13/awesome-cad-hub \
  --create-repo \
  --secrets-file .env.secrets \
  --set-generator-variable \
  --commit \
  --push

What the helper does:

• Creates the GitHub repository with gh repo create when --create-repo is used.
• Writes .github/workflows/daily-update.yml for daily refresh and GitHub Pages deployment.
• Updates awesome.yaml with project.github_url and project.site_url.
• Regenerates README.md and AWESOME.md.
• Removes generated website/ files from the git index.
• Stages only awesome.yaml, .github/workflows/, .gitignore, README.md, AWESOME.md, data/, assets/, and resource/.
• Imports shared repository secrets from .env.secrets when --secrets-file is provided.

Prepare .env.secrets locally and never commit it:

ARK_API_KEY=...
ARK_API_BASE_URL=https://ark.cn-beijing.volces.com/api/coding/v3
ARK_MODEL_NAME=deepseek-v4-flash
SMART_MODEL_NAME=deepseek-v4-pro
MINERU_API_KEY=...
OPENALEX_API_KEY=...
OPENALEX_MAILTO=you@example.com
SEMANTIC_SCHOLAR_API_KEY=...
GH_DISCOVERY_TOKEN=...

For several downstream repositories, you can reuse the same secret file:

for repo in awesome-world-model-hub awesome-AI4CAD-hub awesome-auto-research-hub; do
  gh secret set -R jin-s13/$repo -f .env.secrets
done

The generated repository should use these settings:

Setting	Value
Repository variable	GENERATOR_REPO=jin-s13/awesome-hub-generator
Required secret	ARK_API_KEY
Optional secrets	ARK_API_BASE_URL, ARK_MODEL_NAME, SMART_MODEL_NAME, MINERU_API_KEY, OPENALEX_API_KEY, OPENALEX_MAILTO, SEMANTIC_SCHOLAR_API_KEY, GH_DISCOVERY_TOKEN
Optional variables	ARK_TIMEOUT_SECONDS, ARK_SURVEY_TIMEOUT_SECONDS, ARK_SURVEY_MAX_TOKENS, SEMANTIC_SCHOLAR_REQUEST_INTERVAL_SECONDS

The workflow template in templates/workflows/daily-update.yml runs daily at UTC 00:00, refreshes metadata, rebuilds the site, deploys gh-pages, and commits updated hub data back to main.

Downstream users should read structured metadata from the hub repository's main branch:

data/
assets/
resource/

The gh-pages branch is only the built static website.

Repository Layout

awesome-hub-generator/
├── scripts/
│   ├── build.py                         # Full build pipeline
│   ├── update.py                        # Daily update pipeline
│   ├── hubctl.py                        # Manage local hub checkouts
│   ├── serve_hub.py                     # Pull and run a local Astro dev server
│   ├── paper_sources.py                 # Source aggregation
│   ├── ingest_source.py                 # Markdown/YAML upstream awesome parsing
│   ├── paper_rank.py                    # Explainable read-first ranking
│   ├── fetch_teasers.py                 # Teaser image and PDF figure recovery
│   ├── refresh_interpretations_parallel.py
│   └── literature_survey.py             # Topic synthesis and research analysis
├── templates/
│   ├── astro-site/                      # Static website template
│   └── workflows/daily-update.yml       # GitHub Actions template
├── awesome.yaml.example                 # Hub config template
├── hubs.yaml                            # Local managed hub registry
└── .local/                              # Ignored local hub checkouts

Pipeline Overview

awesome.yaml
    |
    v
Source discovery
    |-- arXiv
    |-- Hugging Face
    |-- upstream awesome repos
    |-- GitHub projects
    |-- OpenAlex / Semantic Scholar signals
    v
Candidate pool -> relevance filtering -> dedupe
    v
Paper ranking + TLDR + deep analysis
    v
Teaser recovery + dataset/project sync
    v
Literature analysis + trend pages
    v
Astro website + structured metadata

Configuration Highlights

Source Discovery

research:
  sources:
    arxiv: true
    huggingface: true
    upstream_awesome: true
    huggingface_datasets: true

  upstream_awesome:
    repos:
      - BunnySoCrazy/Awesome-Neural-CAD
    auto_discover: true

  auto_discover:
    enabled: true
    max_sources: 10

Ranking

research:
  ranking:
    enabled: true
    weights:
      topical_relevance: 0.25
      citation_impact: 0.15
      graph_prestige: 0.15
      citation_velocity: 0.10
      methodology_quality: 0.15
      reproducibility: 0.15
      recency: 0.05

Teaser Recovery

research:
  teasers:
    workers: 4
    retry_fallbacks: true

Fallback teaser SVGs are treated as unresolved. The fetcher retries arXiv HTML, project pages, direct PDFs, CVF/NeurIPS landing-page PDF derivation, MinerU, and local PDF rendering. Remaining fallbacks produce explicit warning logs so they can be investigated later.

Development

Run focused tests:

uv run pytest tests/test_hubctl.py tests/test_serve_hub.py
uv run pytest tests/test_paper_sources.py tests/test_fetch_teasers.py

Check all tests:

uv run pytest

Design Principles

• The generator is a tool, not a concrete hub.
• Generated hubs are independent repositories, not submodules.
• Metadata should stay machine-readable and easy to reuse.
• Warnings should expose unresolved data quality issues instead of hiding them.
• Daily automation should be recoverable, incremental, and safe to rerun.

License

MIT