Scene_Genie - AI-Powered 3D Scene Generation Plugin for Unreal Engine 5

Overview

Scene_Genie is an advanced Unreal Engine 5 plugin that integrates Large Language Models (LLMs) to intelligently generate and arrange 3D scenes. It leverages AI to understand asset properties, scene composition, and user intentions to create compelling 3D environments through natural language prompts.

Key Features

🤖 Multi-LLM Support

• Claude 4 Sonnet & Thinking: Anthropic's latest reasoning models
• Gemini 2.5 Pro: Google's advanced multimodal AI
• GPT-4.1: OpenAI's powerful language model
• OpenAI O3 High: OpenAI's latest reasoning model

🎯 Intelligent Asset Analysis

• Visual Properties: Automatic extraction of colors, materials, and visual style
• Semantic Understanding: Context-aware asset classification and tagging
• Thumbnail Generation: Native UE5 thumbnail system integration

🌍 Advanced Scene Context

• Scene Composition: Spatial analysis and density mapping
• Environment Detection: Indoor/outdoor classification and style recognition
• Lighting Analysis: Comprehensive lighting setup evaluation
• Performance Monitoring: Real-time performance impact assessment

⚡ Action Execution System

• Asset Placement: Intelligent positioning
• Transformation Control: Rotation, scaling, and alignment operations
• Material Application: Dynamic material assignment
• Lighting Setup: Automated lighting configuration
• Distribution Patterns: Procedural object distribution

🎨 Enhanced UI Experience

• Drag & Drop: Intuitive asset handling with native UE5 thumbnails
• Chat Interface: Natural language interaction with AI
• Real-time Feedback: Live action execution updates
• Settings Panel: API key management and configuration

Installation

Prerequisites

• Unreal Engine 5.6 or later
• Valid API keys for desired LLM providers

Setup Instructions

1. Download & Install

   # Clone the repository
   git clone https://github.com/Klakocik/Scene_Genie.git
   
   # Copy to your project's Plugins folder
   cp -r Scene_Genie /path/to/your/project/Plugins/

   You will need to rebuild the project with new plugin added.

2. Configure API Keys Set environment variables for your preferred LLM providers:

   # Windows
   set ANTHROPIC_API_KEY=your_anthropic_key
   set OPENAI_API_KEY=your_openai_key
   set GOOGLE_API_KEY=your_google_key
   
   # macOS/Linux
   export ANTHROPIC_API_KEY=your_anthropic_key
   export OPENAI_API_KEY=your_openai_key
   export GOOGLE_API_KEY=your_google_key

3. Enable Plugin

   • Open your Unreal Engine project
   • Go to Edit → Plugins
   • Search for "Scene_Genie"
   • Enable the plugin and restart the editor

4. Access the Plugin

   • Window → Scene Genie

Usage Guide

🎬 Quick Demo

Intuitive drag and drop functionality - easily add assets from Content Browser to Scene_Genie panel

Basic Workflow

1. Prepare Assets

   • Drag and drop 3D assets into the Scene_Genie panel
   • Assets are automatically analyzed for properties and context

2. Select LLM Model

   • Choose your preferred AI model from the dropdown
   • Each model has different strengths (creativity, reasoning, speed)

3. Write Natural Language Prompt

   "Create a cozy living room scene with a sofa facing a fireplace, 
   add some ambient lighting and place decorative items on shelves"
   

4. Execute Generation

   • Click "Send" to process your request
   • Watch as the AI analyzes assets and generates actions
   • Real-time feedback shows execution progress

📸 Workflow Screenshots

Prompt Preparation

Scene_Genie interface showing prompt input with attached assets and their thumbnails

Results in Action

Scene after AI processing - assets intelligently placed and arranged based on natural language prompt

Advanced Features

Viewport Integration

• The plugin captures viewport screenshots for vision-capable models (viewport screenshot while pressing send)
• Provides visual context for better scene understanding
• Tracks selected actors and their screen positions

Context-Aware Placement

• Respects indoor/outdoor scene classification
• Maintains architectural style consistency
• Avoids high-density area overcrowding

AI Context & Intelligence

What the AI Sees

Scene_Genie provides comprehensive context to LLM models for intelligent scene generation:

📸 Visual Context

• Viewport Screenshot: 512x512 image of current scene for vision-capable models
• Selected Objects: AI knows which actors you have selected (inside viewport) and can reference them (replace, change properties, delete)
• Screen Positions: Tracks where selected objects appear in the viewport

🎯 Asset Intelligence (assets provided with prompt)

• Smart Thumbnails: UE5 native thumbnails
• Asset Properties: Geometry, materials, performance metrics, and semantic tags
• Usage Context: Indoor/outdoor classification, style matching, scale references

🌍 Scene Understanding

• Camera Context: Position, direction, FOV for spatial awareness (viewport data)
• Environment Analysis: Indoor/outdoor type, architectural style, lighting setup
• Composition Analysis: Object density, focal points, scene balance

🔧 User Intent Recognition

• Contextual Commands: "Replace the selected chair", "move this to the left"
• Spatial References: "behind the building", "next to the lamp"
• Style Consistency: Maintains visual coherence across scene elements

This enables the AI to make intelligent decisions about placement, scale, rotation, and style while respecting performance budgets and scene composition.

Technical Architecture

Core Components

LLM Service (ULLMService)

• Manages communication with different AI providers
• Handles request formatting and response parsing
• Supports vision-capable models with image data

Asset Processor (USceneGenieAssetProcessor)

• Analyzes 3D assets for visual and semantic properties
• Extracts performance metrics and optimization data
• Generates thumbnails using native UE5 system

Action Executor (USceneGenieActionExecutor)

• Executes AI-generated actions in the scene
• Currently supports 9 different action types
• Provides real-time feedback and error handling

Scene Context (FSceneGenieContext)

• Captures comprehensive scene information
• Analyzes composition, lighting, and environment
• Tracks performance state and optimization opportunities

Data Flow

User Input → Asset Analysis → Scene Context → LLM Request → 
Action Generation → Action Execution → Scene Updates

API Reference

Supported Actions

Action	Description	Parameters
place_asset	Places an asset in the scene	asset_path, location, rotation, scale
rotate_asset	Rotates an existing asset	asset_id, rotation, relative
set_scale	Scales an asset	asset_id, scale
group_assets	Groups multiple assets	asset_ids, group_name
add_lighting	Adds lighting to scene	light_type, location, intensity, color
apply_material	Applies material to asset	asset_id, material_path, slot_index
align_to_surface	Aligns asset to surface	asset_id, surface_normal
distribute_in_bounds	Distributes assets in area	asset_paths, bounds_min, bounds_max, count
delete_asset	Removes asset from scene	asset_id

Example Prompts

Complex Scene:

"Create a medieval tavern interior with wooden tables, chairs, 
barrels, and warm lighting. Make it feel cozy and lived-in."

Style-Specific:

"Arrange these sci-fi assets to create a futuristic laboratory 
with blue accent lighting and clean geometric layout"

Configuration

Plugin Settings

The plugin can be configured through environment variables or the settings panel:

Note: Let's be real, use environment variables

// API Configuration
ANTHROPIC_API_KEY=your_key_here
OPENAI_API_KEY=your_key_here
GOOGLE_API_KEY=your_key_here

### Model Configuration

Each LLM model has specific parameters:

```cpp
FLLMModelConfig ClaudeConfig(
    ELLMModel::Claude4Sonnet,
    ELLMProvider::Anthropic,
    "claude-sonnet-4-20250514",
    "Claude Sonnet 4.0",
    "https://api.anthropic.com/v1/messages",
    0.6f,   // Temperature
    0.9f,   // Top-P
    16384,  // Max Tokens
    10000   // Thinking Budget
);

Troubleshooting

Common Issues

API Key Issues:

Error: "API key not found"
Solution: Set environment variables or use settings panel

Debug Logging

The plugin includes comprehensive logging enabled by default. Check the Unreal Engine Output Log for Scene Genie messages at various verbosity levels (Log, Display, Warning, VeryVerbose). Filter logs by LogTemp.

Contributing

Development Setup

1. Clone Repository

   git clone https://github.com/yourusername/Scene_Genie.git
   cd Scene_Genie

2. Build Plugin

   # Generate project files
   UnrealBuildTool.exe -projectfiles -project="YourProject.uproject"
   
   # Build in development mode
   UnrealBuildTool.exe YourProject Development Win64

3. Code Style

   • Follow Unreal Engine coding standards
   • Use proper UPROPERTY/UFUNCTION macros
   • Include comprehensive documentation

Contribution Guidelines

• Submit pull requests with clear descriptions
• Update documentation for API changes
• Follow semantic versioning

To-Do List

To-Do

• Scene description prompt system refactor
• Add more details on global assets rotation (default) to prompts
• Add safety check and confiramtion for assets deleted by plugin

Done

• Interactive plugin window
• Drag-drop assets from content browser
• Assets thumbnails like in content browser
• Enforcing the response JSON schema from LLM
• LLM's use only given tools with provided assets
• Thumbnails with metadata provided with prompt
• Executing functions sent by the LLM

License

This project is licensed under the GPL v3 License - see the LICENSE file for details.

Acknowledgments

• Epic Games for Unreal Engine 5
• Anthropic, OpenAI, and Google for LLM APIs
• The Unreal Engine community for inspiration and feedback

Support

Contact

• Author: Krzysztof Marciniak
• Email: krzysztof.marciniak96@gmail.com

---

Scene_Genie - Bringing AI-powered creativity to Unreal Engine 5 🎮✨