AI Video Generation System with Character Consistency

A comprehensive Python-based AI video generation system that processes movie scripts and storyboards to generate videos with consistent characters. Built as a modular, production-ready system with advanced character consistency techniques.

🎬 Features

• Script Analysis: Advanced NLP-based character extraction from scripts using spaCy
• Character Generation: Text-to-image generation for character reference sheets
• Storyboard Analysis: Computer vision for character identification in storyboards
• Video Generation: Text-to-video with character consistency using multiple AI models
• Database Management: SQLite-based persistence for projects, characters, and videos
• CLI Interface: Easy-to-use command-line interface for all operations
• Character Consistency: Advanced techniques including reference image injection and LoRA fine-tuning

🏗️ System Architecture

Core Modules:

1. ScriptParser - NLP-based character extraction from scripts
2. CharacterGenerator - Text-to-image generation for character references
3. StoryboardAnalyzer - Computer vision for character identification in storyboards
4. VideoGenerator - Text-to-video with character consistency
5. DatabaseManager - Character and scene data management
6. APIManager - External API integrations

🚀 Quick Start

Prerequisites

• Python 3.9+
• CUDA-compatible GPU (recommended for faster processing)
• 8GB+ RAM (16GB+ recommended)

Installation

1. Clone the repository

git clone <repository-url>
cd ai_video_generator

2. Install Python dependencies

pip install -r requirements.txt

3. Download spaCy model

python -m spacy download en_core_web_sm

4. Download NLTK data

python -c "import nltk; nltk.download('punkt'); nltk.download('vader_lexicon')"

5. Set up environment variables

cp env.example .env
# Edit .env with your API keys

Basic Usage

1. Create a new project

python src/main.py create-project --name "My Movie" --script "script.txt"

2. Extract characters from script

python src/main.py extract-characters --project-id 1

3. Generate character reference images

python src/main.py generate-references --project-id 1

4. Analyze storyboard

python src/main.py analyze-storyboard --project-id 1 --storyboard "storyboard.png"

5. Generate video

python src/main.py generate-video --project-id 1

📋 Detailed Usage

1. Script Parser Module

Extracts characters, scenes, and dialogue from screenplay formats.

from src.script_parser import ScriptParser

parser = ScriptParser()
characters, scenes = parser.parse_script("script.txt")

# Get character information
for char in characters:
    print(f"Character: {char.name}")
    print(f"Importance: {char.importance_score}")
    print(f"Traits: {char.traits}")

Features:

• Supports Final Draft, PDF, and TXT formats
• Extracts character descriptions and traits using spaCy NER
• Ranks characters by importance based on dialogue frequency
• Identifies character relationships and emotions

2. Character Generator Module

Generates consistent character reference images using text-to-image models.

from src.character_generator import CharacterGenerator

generator = CharacterGenerator()
generator.load_models()

character_info = {
    'id': 'char_001',
    'name': 'John Smith',
    'description': 'A confident businessman in his 40s',
    'traits': {'confident': 3, 'intelligent': 2}
}

reference = generator.generate_character_views(character_info, "./output")

Features:

• Multiple character views (front, side, 3/4, expressions)
• Character consistency techniques (ControlNet, LoRA)
• Batch processing for multiple characters
• Reference sheet generation

3. Storyboard Analyzer Module

Analyzes storyboard images to identify characters using computer vision.

from src.storyboard_analyzer import StoryboardAnalyzer

analyzer = StoryboardAnalyzer()
analyzer.load_character_database(characters)

frames = analyzer.analyze_storyboard("storyboard.png")
character_matches = analyzer.identify_characters_in_scene(frames)

Features:

• Automatic frame detection in storyboard sheets
• Face recognition using DeepFace and face_recognition
• Character matching with confidence scores
• Multiple detection methods (contours, grid, text-based)

4. Video Generator Module

Generates videos from storyboard scenes with character consistency.

from src.video_generator import VideoGenerator

generator = VideoGenerator()
generator.load_models()

video = generator.generate_scene_video(scene_info, character_matches, "./output")

Features:

• Multiple video generation backends (Open-Sora, RunwayML, HuggingFace)
• Character consistency maintenance
• Batch video generation
• Quality assessment

🔧 Configuration

Environment Variables

Create a .env file with the following variables:

# API Keys
OPENAI_API_KEY=your_openai_api_key_here
HUGGINGFACE_API_KEY=your_huggingface_token_here
RUNWAYML_API_KEY=your_runwayml_api_key_here

# Model Configuration
DEFAULT_VIDEO_MODEL=open-sora
DEFAULT_IMAGE_MODEL=stabilityai/stable-diffusion-xl-base-1.0
DEFAULT_FACE_MODEL=Facenet

# Processing Configuration
MAX_CONCURRENT_VIDEOS=2
MAX_CONCURRENT_IMAGES=4
VIDEO_QUALITY=high
IMAGE_RESOLUTION=1024

Model Configuration

The system supports multiple AI models:

Image Generation:

• Stable Diffusion XL
• DALL-E 3 (via API)
• Midjourney (via API)

Video Generation:

• Open-Sora (local)
• OpenAI Sora (API)
• RunwayML (API)
• HuggingFace models (API)

Face Recognition:

• DeepFace (Facenet, VGG-Face, ArcFace)
• face_recognition library

📊 Database Schema

The system uses SQLite for data persistence with the following tables:

• projects: Project information and status
• characters: Character data and reference images
• scenes: Scene information from scripts
• scene_characters: Character-scene relationships
• videos: Generated video metadata

🎯 Advanced Features

Character Consistency Techniques

1. Reference Image Injection: Pass character reference images to video generation
2. LoRA Fine-tuning: Train character-specific adapters for Stable Diffusion
3. Embedding Consistency: Use consistent text embeddings for characters
4. Multi-shot Character Generation: Advanced character consistency across scenes

Computer Vision Enhancements

1. Multi-modal Matching: Combine face recognition with clothing/pose matching
2. Confidence Thresholding: Only proceed with high-confidence character matches
3. Manual Override: UI for correcting mis-identified characters

Video Generation Optimizations

1. Batch Processing: Generate multiple scenes in parallel
2. Quality Control: Automatic filtering of poor-quality outputs
3. Style Consistency: Maintain visual style across scenes

🧪 Testing

Run the test suite:

# Run all tests
pytest tests/

# Run with coverage
pytest --cov=src tests/

# Run specific test file
pytest tests/test_script_parser.py

📁 Project Structure

ai_video_generator/
├── src/
│   ├── script_parser.py          # Script analysis and character extraction
│   ├── character_generator.py    # Character reference image generation
│   ├── storyboard_analyzer.py    # Storyboard analysis and character identification
│   ├── video_generator.py        # Video generation with character consistency
│   ├── database.py               # Database management and models
│   ├── config.py                 # Configuration management
│   └── main.py                   # CLI application
├── data/
│   ├── projects/                 # Project files
│   ├── characters/               # Character reference images
│   └── videos/                   # Generated videos
├── models/                       # Local AI models
├── tests/                        # Test suite
├── requirements.txt              # Python dependencies
├── env.example                   # Environment variables template
└── README.md                     # This file

🚨 Troubleshooting

Common Issues

1. Model Loading Errors

   • Ensure you have sufficient RAM (8GB+)
   • Check CUDA installation for GPU acceleration
   • Verify model downloads completed successfully

2. API Rate Limits

   • Adjust MAX_CONCURRENT_VIDEOS and MAX_CONCURRENT_IMAGES in config
   • Implement exponential backoff for API calls

3. Character Recognition Issues

   • Ensure character reference images are high quality
   • Adjust confidence thresholds in storyboard analyzer
   • Use manual override for difficult cases

4. Video Generation Failures

   • Check API key validity
   • Verify sufficient disk space
   • Monitor GPU memory usage

Performance Optimization

1. GPU Memory Management

   • Use smaller batch sizes
   • Enable gradient checkpointing
   • Clear CUDA cache between operations

2. API Optimization

   • Implement request caching
   • Use connection pooling
   • Batch API requests when possible

🤝 Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests for new functionality
5. Submit a pull request

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

• OpenAI for Sora and DALL-E APIs
• HuggingFace for model hosting and APIs
• Stability AI for Stable Diffusion models
• The open-source community for various AI libraries

📞 Support

For support and questions:

• Create an issue on GitHub
• Check the troubleshooting section
• Review the documentation

---

Note: This system requires significant computational resources and API access. Ensure you have appropriate hardware and API keys before running.