Transform any content into beautiful AI-powered wikis
An intelligent wiki generator that transforms books, websites, and documents into comprehensive, searchable wiki sites with automatically extracted entities, relationships, and beautiful formatting.
- 📄 Multi-Format Support - PDFs, websites, plain text, and markdown
- 🤖 AI-Powered Extraction - Automatic entity and relationship extraction using LLMs
- 🔄 Multi-Provider LLM - Support for Anthropic Claude and OpenAI with automatic fallback
- 🎨 Beautiful Output - Fandom-style static sites using MkDocs Material theme
- 🔗 Smart Linking - Automatic cross-linking between related entities
- 💾 Local Database - SQLite storage for all extracted data
- 🛡️ Robust Architecture - Retry logic, error handling, and graceful fallback
The system extracts and generates wiki articles for:
- 👤 Characters - People, protagonists, supporting roles
- 🗺️ Locations - Cities, buildings, regions, landmarks
- 🏛️ Organizations - Groups, companies, factions, institutions
- 💡 Concepts - Ideas, theories, systems, technologies
- ⚔️ Events - Major occurrences, battles, turning points
- ⚡ Items - Significant objects, artifacts, weapons
- Python 3.10 or higher
- uv (dependency management)
- API key for Anthropic Claude or OpenAI
# Clone the repository (if from git)
cd wiki-generator
# Install uv if you haven't already
curl -LsSf https://astral.sh/uv/install.sh | sh
# Install dependencies with uv
uv sync --extra dev
# Set up environment variables
cp .env.example .env
# Edit .env and add your API keys:
# ANTHROPIC_API_KEY=sk-ant-your-key-here
# or
# OPENAI_API_KEY=sk-your-key-hereuv run wiki-generator check-configTry the included sample story:
uv run wiki-generator generate examples/sample_story.txt --output ./my-wiki --verboseThen serve it locally:
uv run wiki-generator serve ./my-wikiOpen http://localhost:8000 to view your wiki!
# Generate from a PDF
wiki-generator generate book.pdf --output ./book-wiki
# Generate from a website
wiki-generator generate https://example.com/article --output ./web-wiki
# Generate and serve immediately
wiki-generator generate source.txt --serve
# Use custom configuration
wiki-generator generate book.pdf --config custom-config.yaml
# Check configuration and API keys
wiki-generator check-configwiki-generator generate [SOURCE] [OPTIONS]
Arguments:
SOURCE Path to PDF, URL, or text file
Options:
-o, --output DIR Output directory (default: ./output/site)
-c, --config FILE Configuration file path
-v, --verbose Enable verbose logging
-s, --serve Serve wiki after generation
--help Show help message┌─────────────┐
│ Source │ (PDF, Web, Text)
└──────┬──────┘
│
┌──────▼──────────┐
│ Content Parser │ Extract text and metadata
└──────┬──────────┘
│
┌──────▼──────────┐
│ Entity Extractor│ AI-powered entity extraction
└──────┬──────────┘
│
┌──────▼──────────┐
│ Wiki Generator │ Create comprehensive articles
└──────┬──────────┘
│
┌──────▼──────────┐
│ Site Builder │ MkDocs static site generation
└──────┬──────────┘
│
▼
Beautiful Wiki
- Content Parsers: Handle PDF, web, and text sources
- LLM Orchestrator: Multi-provider support with automatic fallback
- Entity Extractor: Identifies and structures entities using AI
- Wiki Generator: Creates detailed, cross-linked articles
- Site Builder: Generates beautiful, searchable static sites
- Storage Layer: SQLite database for persistence
The system uses config/default_config.yaml. Key settings:
llm:
providers:
- name: anthropic
model: claude-3-5-sonnet-20241022
- name: openai
model: gpt-4-turbo-preview
fallback_order:
- anthropic
- openai
extraction:
chunk_size: 4000
entity_types:
- character
- location
- organization
- concept
- event
- item
site:
theme: material
site_name: Generated WikiCreate your own YAML file to override defaults:
# my-config.yaml
site:
site_name: "My Fantasy World Wiki"
site_description: "Complete guide to my story universe"
extraction:
chunk_size: 6000
min_confidence: 0.7Use it with:
wiki-generator generate source.txt --config my-config.yamlFrom the included examples/sample_story.txt, the generator creates:
📁 my-wiki/
├── 📄 index.html (home page with statistics)
├── 👤 characters/
│ ├── Queen Celestia.md
│ ├── Commander Theron Blackwood.md
│ ├── Elara Moonwhisper.md
│ └── ...
├── 🗺️ locations/
│ ├── Kingdom of Elendril.md
│ ├── Silverhaven.md
│ ├── Whispering Woods.md
│ └── ...
├── 🏛️ organizations/
│ ├── Shadow Council.md
│ ├── Royal Guard.md
│ └── ...
└── ⚡ items/
├── Starlight Amulet.md
├── Shadowbane.md
└── ...
Each article includes:
- Comprehensive description
- Related entities (automatically linked)
- Type-specific sections
- Metadata and tags
- Cross-references
# Run all tests
uv run pytest
# Run with verbose output
uv run pytest -v
# Run specific test file
uv run pytest tests/test_models.pyCurrent test coverage:
- ✅ Core data models
- ✅ Configuration loading
- ⏳ Integration tests (pending)
- ⏳ Parser tests (pending)
- GETTING_STARTED.md - Comprehensive setup and usage guide
- ARCHITECTURE.md - System design and technical details
- PROJECT_SUMMARY.md - Implementation status and features
- CHECKLIST.md - Development progress tracking
wiki-generator/
├── src/wiki_generator/ # Main package
│ ├── core/ # Models, config, pipeline
│ ├── llm/ # LLM providers and orchestration
│ ├── parsers/ # Content parsers
│ ├── extraction/ # Entity extraction
│ ├── generation/ # Wiki generation
│ ├── site/ # Static site building
│ └── storage/ # Database layer
├── config/ # Configuration files
│ ├── default_config.yaml
│ └── prompts/ # Jinja2 templates
├── tests/ # Unit tests
├── examples/ # Sample content
└── docs/ # Documentation
Contributions are welcome! Areas for improvement:
- Image Pipeline - Implement automatic image acquisition
- Integration Tests - End-to-end testing
- Performance - Optimization for large books
- Local LLMs - Support for Ollama, llama.cpp
- UI Enhancements - Custom themes, visualizations
-
Images: Automatic image acquisition not yet implemented (Phase 6)
- Articles won't have images automatically
- Can manually add images to markdown files
-
Performance: Optimized for books up to ~200 pages
- Larger books work but may take longer
- Consider chunking very large content
-
LLM Dependency: Requires API access to Claude or GPT
- Local LLM support planned
- API costs apply (typically $1-3 for a medium book)
Typical processing times (with Claude Sonnet 4.5):
- Short story (10 pages): 2-5 minutes
- Medium book (100 pages): 15-30 minutes
- Large book (200 pages): 45-90 minutes
Cost estimates:
- Small: ~$0.10-0.30
- Medium: ~$1-3
- Large: ~$3-10
- Image extraction from PDFs
- Web image search
- AI image generation (DALL-E)
- Integration tests
- Performance optimization
- Local LLM support (Ollama)
- Multi-language wikis
- Custom entity types
- Relationship visualization
- Incremental updates
- Web UI
- Collaborative editing
- Export to Notion/Obsidian
- GitHub Pages auto-deploy
- Docker deployment
Built with excellent open-source tools:
- Anthropic Claude - AI entity extraction
- OpenAI GPT-4 - Alternative LLM provider
- MkDocs Material - Beautiful documentation theme
- Instructor - Structured LLM outputs
- Pydantic - Data validation
- uv - Fast Python package manager
MIT License - see LICENSE file for details
- 📖 Read GETTING_STARTED.md for detailed setup instructions
- 🏗️ See ARCHITECTURE.md for technical details
- 🐛 Report issues on GitHub
- 💬 Enable
--verboseflag for debugging
Made with ❤️ using AI and Python
Transform your content into beautiful wikis today! 🚀