Bloom (SJHacks2026)

Bloom is a web-based creative IDE that turns sketches, voice, and simple descriptions into 3D models.

Built with accessibility in mind, Bloom removes the need for complex 3D tools—users can draw, speak, and describe their ideas, while the app handles the heavy lifting and guides them through the process.

Core Capabilities

• Sketch → 3D — draw on a 2D canvas and generate .glb assets in one step
• Voice + Chat Input — use speech-to-text or natural language to refine your model
• Interactive 3D Workspace — preview, rotate, and transform models in real time
• Context-Aware AI — get suggestions based on your current sketch and workflow

Devpost: View our project on Devpost

Project Structure

SJHacks2026/
├── frontend/   # Next.js app (landing page, workspace UI, AI chat route)
├── backend/    # FastAPI app (sketch-to-3D pipeline + API routes)
├── README.md
└── RUN_APP.md

Tech Stack

• Frontend: Next.js 16, React 19, TypeScript, Tailwind, Fabric.js, Three.js (@react-three/fiber + @react-three/drei), Zustand
• Backend: FastAPI, Pydantic, HTTPX, Uvicorn, Python dotenv
• AI/Generation:
  • Meshy Image-to-3D API (backend)
  • Google Gemini (frontend chat route)

Features

• Sketchpad with clear/toggle prediction controls
• One-click sketch compile to 3D model (Compile & Generate)
• 3D model viewport with transform gizmo and generation metadata
• Fallback behavior to local mock model if Meshy is unavailable
• Context-aware AI chat that can inspect current sketch data URL

Prerequisites

• Node.js 20+
• npm
• Python 3.11+ (3.13 also works in this repo)

Quick Start

1) Clone and install dependencies

git clone <your-repo-url>
cd SJHacks2026

Backend dependencies:

cd backend
pip install -r requirements.txt

Frontend dependencies:

cd ../frontend
npm install

2) Configure environment variables

Create backend/.env (you can copy backend/.env.example):

MESHY_API_KEY=your_meshy_api_key
# Optional
# MESHY_AI_MODEL=meshy-6
# SKETCH_TO_3D_PROVIDER=meshy
# SKETCH_TO_3D_POLL_INTERVAL=3
# SKETCH_TO_3D_POLL_TIMEOUT=300

Create frontend/.env.local:

# Empty value means same-origin; set this when backend runs on another origin.
NEXT_PUBLIC_API_BASE_URL=http://127.0.0.1:8000

# Required for AI chat route
GEMINI_API_KEY=your_gemini_api_key
# Optional
# GEMINI_MODEL=gemini-1.5-flash

3) Run backend

From repo root:

uvicorn backend.main:app --host 127.0.0.1 --port 8000

Health check:

curl http://127.0.0.1:8000/health

Expected response:

{"status":"ok"}

4) Run frontend

In a second terminal:

cd frontend
npm run dev

Open:

• http://localhost:3000 (landing page)
• http://localhost:3000/workspace (main app)

How Generation Works

1. Frontend exports sketch canvas as PNG data URL.
2. POST /api/sketch-to-3d sends sketch to backend.
3. Backend pipeline:
   • decodes base64
   • writes a temp .png
   • sends to Meshy Image-to-3D
   • polls until completion
   • optionally mirrors returned .glb into backend/static/generated/
4. Frontend loads returned glb_url in the 3D viewport.

If Meshy fails or no API key is configured, backend falls back to backend/static/mock_model.glb.