Note: This repository contains the Machine Learning pipeline, model training scripts, and the backend inference server. The frontend chat interface (TypeScript/Lovable) for this project can be found here.
short_generation.mp4
*Note: The generation process in the demo video is sped up 60x. Inference was executed locally on consumer hardware (laptop GPU). Since the model uses auto-regressive next-token prediction to generate up to 4096 tokens per mesh, real-time generation requires dedicated compute cluster setups.*
🎥 Watch the full 2-minute demo video here
This project is my graduation thesis, exploring the direct generation of structured 3D assets (.obj format) from natural language prompts using Large Language Models. Instead of using diffusion models, this pipeline treats 3D mesh generation as an auto-regressive next-token prediction task.
- Base Model: LLaMa 3.1 8B Instruct
- Fine-Tuning: Parameter-efficient supervised fine-tuning using QLoRA (8-bit, r=64) to accommodate constrained GPU infrastructure.
- Context Limit: 4096 tokens (generating approximately 250 polygons per response).
Figure 1: Tokenization of 3D meshes for Auto-Regressive Language Modeling.
Preparing raw spatial data for LLM consumption was the most intensive part of the pipeline. I created a custom dataset of 13,000 high-quality 3D objects starting from the Objaverse-XL dataset.
Preprocessing Pipeline:
- Filtering: Polygon count reduction (<500 polygons) to fit within LLM token limits.
- Computational Geometry: Automated correction of surface normals and missing polygons.
- Quantization: Vertex coordinate quantization to map continuous 3D spatial data into a discrete token vocabulary.
- Augmentation: Rotations and dynamic name sampling for robust prompt alignment.
Figure 2: End-to-end data processing and filtering pipeline.
To serve the model in a real-time chat interface, I engineered a high-performance backend:
- Parallel Generation: Implemented a multi-processing pool with isolated child processes to handle concurrent user requests without blocking.
- Token Streaming: Utilized Server-Sent Events (SSE) to stream generated OBJ tokens to the frontend in real time.
- Storage: PostgreSQL for chat history and AWS S3 (via presigned URLs) for long-term 3D model storage.
Figure 3: Backend architecture.
As an R&D project running on constrained compute, the final output quality is heavily stylized and noisy. I conducted an analysis of these limitations, which provides clear directions for future architectural improvements:
- Lack of Inductive Bias: Auto-regressive LLMs treat all tokens equally. Unlike specialized 3D architectures, standard cross-entropy loss lacks a metric error function (e.g., Chamfer distance) to penalize geometric deviations.
- Dataset Size: 13,000 examples are insufficient for an 8B model to deeply generalize spatial structures from scratch.
- LoRA Bottleneck: Constraining trainable parameters via LoRA limits the model's capacity to fundamentally restructure its internal representation for 3D topology.
This project is an applied implementation and evaluation inspired by the following research:
- Llama-mesh: Wang Z. et al. "Llama-mesh: Unifying 3d mesh generation with language models" (2024).
Tech Stack: PyTorch, Open3D, Transformers, peft (QLoRA), FastAPI, PostgreSQL, AWS S3, Weights & Biases.