Skip to content

INDUSTRY_BENCHMARKS.md

Latest commit

 

History

History
59 lines (45 loc) · 3.55 KB

INDUSTRY_BENCHMARKS.md

File metadata and controls

59 lines (45 loc) · 3.55 KB
Raw

🚀 DreamBees MLX Metal Industry Benchmark Report

Published: 2026-08-08
Hardware Substrate: Apple Silicon Metal GPU (darwin arm64)
Daemon Substrate: electron/mlx/mlx_image_daemon.py
Test Suite: 12 Standardized Real-World Multi-Category Prompts

📊 Executive Multi-Model Performance Summary

Metric FLUX.2 Klein (4B) Z-Image Turbo (6B) Benchmark Standard
Total Test Cases 12 / 12 12 / 12 Standardized Multi-Category Matrix
Success Rate 100.0% 100.0% 100% Zero-Crash Target
Total Suite Duration 931.25s 542.84s Automated Multi-Model Sweep
Average Test Latency 77,604 ms 45,236 ms Real-World Sub-Minute DiT Inference
Peak Active VRAM ~3,766 MB ~5,416 MB Dynamic Model VRAM Auto-Scaler
TAESD Preview Latency < 1.0 ms < 1.0 ms Sub-Millisecond Latent Projection
IPC Stdio Transport < 0.5 ms < 0.5 ms Zero-GC Monomorphic JSON-RPC

🏎️ Industry Category Benchmark Matrix

# Benchmark Name Category Resolution FLUX.2 Klein (4B) Z-Image Turbo (6B) Status
1 Zaha Hadid Biomorphic Oasis Organic Architecture 512x512 81293 ms 49512 ms ✅ PASS
2 Iridescent Mechanical Hummingbird Cybernetic Macro Fauna 512x512 76871 ms 44994 ms ✅ PASS
3 Deep Space Observatory Deck Sci-Fi Interior 512x512 76895 ms 44927 ms ✅ PASS
4 Sunlit Lavender Field Impasto Impressionist Fine Art 512x512 76902 ms 44922 ms ✅ PASS
5 High-Fashion Avant-Garde Portrait Fashion & Studio Optics 512x512 76874 ms 44947 ms ✅ PASS
6 Neon Noodle Shop Diorama Isometric 3D Diorama 512x512 76967 ms 44853 ms ✅ PASS
7 Bioluminescent Quartz Cave Mineral Subsurface Optics 512x512 77343 ms 44801 ms ✅ PASS
8 Sunlit Terracotta Courtyard Minimalist Vector & Shadows 512x512 77712 ms 44761 ms ✅ PASS
9 Layered Origami Mountain Forest Papercraft & Origami Art 512x512 77606 ms 44777 ms ✅ PASS
10 Quantum Hive Drone Swarm Kinetic Sci-Fi Dynamics 512x512 77585 ms 44779 ms ✅ PASS
11 Cybernetic Glassmorphism Interface UI & Glassmorphism Optics 512x512 77597 ms 44774 ms ✅ PASS
12 Sakura Blossom Samurai Anime & Fluid Watercolor 512x512 77595 ms 44789 ms ✅ PASS

🎨 Visual Benchmark Image Outputs

Outputs for each model card are archived in:

  • tests/benchmarks/outputs/flux2-klein-4b/
  • tests/benchmarks/outputs/z-image-turbo/

🔬 Benchmark Methodology & Architectural Sympathy

  1. Unthrottled Metal Priority: The Python daemon process runs directly on the macOS Metal subsystem with low process niceness (nice -n 10) for 60 FPS WindowServer desktop fluidity.
  2. Dynamic Tier VRAM Limits: configure_mlx_memory_limits auto-scales MLX VRAM limit dynamically based on model parameter count (30% RAM for 4B models, 40% RAM for 6B/9B models), reserving 60%–75% of system hardware RAM for macOS.
  3. Zero-GC Monomorphic IPC: JSON-RPC payloads over stdio use stable key order Hidden Classes for TurboFan inline cache optimization.
  4. Sub-Millisecond Latent Decode: Neural latent preview frames use calibrated multi-model TAESD projection matrices executing in <1ms.

Generated automatically by tests/benchmarks/run_benchmarks.py