Skip to content
BranchesTags

Repository files navigation

FRUST

FRUST Logo

A high-performance, GPU-accelerated LLM inference desktop application

License Tauri Rust Leptos

Features • Installation • Usage • Architecture • Development • Contributing

Overview

FRUST is a desktop application for running Large Language Models (LLMs) locally with GPU acceleration. Built with Tauri 2.0, it combines a Rust backend with a Leptos WASM frontend to deliver fast, efficient inference on consumer hardware.

Key Highlights

  • 🚀 GPU Acceleration - WebGPU compute shaders for matrix operations and attention
  • 💾 Memory Efficient - KV cache sparsification and quantization support (Q4_K_M)
  • 🤖 Multi-Model Support - Qwen 2.5, Qwen 3, Gemma 3, and SmolLM
  • 🖥️ Cross-Platform - Windows, macOS, and Linux support
  • 📦 Self-Contained - No external dependencies required for end users
  • ⚡ CCC Integration - JIT compilation with SSA-based optimizing compiler
  • 📚 Mastery System - Passive learning with RAG pipeline
  • 🔄 Daemon Mode - Background service with self-optimization
  • 🎯 Omnitrix - Automatic form switching based on intent

Features

Model Support

Model Family CPU Inference GPU Inference Quantization
Qwen 2.5 ✅ ✅ Q4_K_M
Gemma 3 ✅ ✅ Q4_K_M
SmolLM ✅ ✅ Q4_K_M

GPU Acceleration

FRUST leverages WebGPU for cross-platform GPU compute:

  • Matrix Multiplication - Optimized WGSL shaders for quantized weights
  • Attention Mechanism - GPU-accelerated self-attention with KV caching
  • Layer Normalization - RMSNorm and LayerNorm on GPU
  • Activation Functions - GELU, SiLU, ReLU on GPU

Memory Management

  • Model Weights: Stored in CPU RAM (~1GB for 1.5B model with Q4_K_M)
  • KV Cache: GPU VRAM with sparsification support (~336MB for 1.5B model)
  • Compute Buffers: ~385MB VRAM for intermediate activations

CCC Integration (JIT Compilation)

FRUST includes a complete SSA-based optimizing compiler for JIT execution of LLM-generated C code:

  • SSA Intermediate Representation - Arena-allocated IR with phi nodes
  • Optimization Passes - DCE, constant propagation, CSE, constant folding
  • Register Allocation - Linear scan with spill handling
  • Native Codegen - x86_64 machine code emission
  • In-Memory Execution - No external processes, no temp files

Mastery System (Passive Learning)

Continuous knowledge acquisition with RAG pipeline:

  • Topic Registry - Built-in topics (Rust, Python, CUDA, ML, Web Dev)
  • RAG Pipeline - Knowledge chunking and embedding storage
  • Progress Tracking - Semantic density measurement
  • Research Crawler - Automated knowledge acquisition

Daemon Mode

Background service with system-level control:

  • System Telemetry - CPU, RAM, GPU temp, VRAM monitoring
  • Process Monitoring - Game detection, resource classification
  • Memory Swap - Model tiering (VRAM → RAM → Disk)
  • Self-Optimization - Automatic resource management

Omnitrix (Form Switching)

Automatic form switching based on user intent:

  • Chat Form - General conversation and Q&A
  • Coder Form - Code generation and debugging
  • Tool Form - Web search and file operations
  • Crawler Form - Research and knowledge acquisition

Installation

Prerequisites

  • Rust 1.70 or later
  • Node.js 18+ (for frontend tooling)
  • CUDA Toolkit 12.x (optional, for NVIDIA GPU acceleration)

Build from Source

# Clone the repository
git clone https://github.com/Zucloak/FRUST.git
cd FRUST

# Install frontend dependencies
cd frontend && npm install && cd ..

# Build and run in development mode
cargo tauri dev

Build for Production

# Build release binary
cargo tauri build

# The executable will be in src-tauri/target/release/

Feature Flags

Configure hardware acceleration in src-tauri/Cargo.toml:

# Default: WebGPU (works on all GPUs)
default = ["wgpu"]

# NVIDIA CUDA support (requires CUDA Toolkit)
cuda = ["candle-core/cuda", "candle-nn/cuda", "candle-transformers/cuda"]

# Flash Attention 2 (requires RTX 3000+ or A100)
cuda-flash = ["cuda", "candle-flash-attn"]

# Apple Metal support
metal = ["candle-core/metal", "candle-nn/metal", "candle-transformers/metal"]

Usage

Starting the Application

  1. Launch FRUST
  2. Select a model from the dropdown (models are auto-downloaded from HuggingFace)
  3. Configure generation parameters (temperature, top-p, max tokens)
  4. Start chatting!

Model Management

Models are automatically downloaded to your HuggingFace cache directory:

  • Windows: C:\Users\<user>\.cache\huggingface\hub\
  • macOS/Linux: ~/.cache/huggingface/hub/

Keyboard Shortcuts

Shortcut Action
Enter Send message
Shift + Enter New line
Escape Stop generation

Architecture

FRUST follows a three-tier architecture:

┌─────────────────────────────────────────────────────────────┐
│                     Leptos Frontend (WASM)                  │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────────────┐  │
│  │   Chat.rs   │  │ Settings.rs │  │    Tauri Bridge     │  │
│  └─────────────┘  └─────────────┘  └─────────────────────┘  │
└─────────────────────────────────────────────────────────────┘
                              │
                              │ Tauri IPC
                              ▼
┌─────────────────────────────────────────────────────────────┐
│                    Rust Backend (Tauri)                     │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────────────┐  │
│  │   State.rs  │  │ Inference.rs│  │    Model Loaders    │  │
│  └─────────────┘  └─────────────┘  └─────────────────────┘  │
└─────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│                    WebGPU Compute Engine                    │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────────────┐  │
│  │  Device.rs  │  │  Tensor.rs  │  │   GPU KV Cache      │  │
│  └─────────────┘  └─────────────┘  └─────────────────────┘  │
└─────────────────────────────────────────────────────────────┘

Key Components

Component Location Description
Frontend frontend/src/ Leptos-based UI components
Backend src-tauri/src/ Tauri commands and state management
WebGPU src-tauri/src/wgpu_backend/ GPU compute operations
Models src-tauri/src/models/ Model implementations
Shaders src-tauri/src/wgpu_backend/shaders/ WGSL compute shaders

For detailed architecture documentation, see docs/ARCHITECTURE.md.

Development

Project Structure

frust/
├── frontend/                 # Leptos WASM frontend
│   ├── src/
│   │   ├── app.rs           # Main application component
│   │   ├── components/      # UI components
│   │   ├── tauri_bridge.rs  # Tauri IPC bindings
│   │   └── storage/         # Local storage utilities
│   ├── index.html
│   └── Trunk.toml
├── src-tauri/                # Rust backend
│   ├── src/
│   │   ├── main.rs          # Application entry point
│   │   ├── state.rs         # Application state
│   │   ├── inference.rs     # Inference orchestration
│   │   ├── models/          # Model implementations
│   │   ├── tokenizers/      # Tokenizer implementations
│   │   └── wgpu_backend/    # WebGPU compute
│   ├── Cargo.toml
│   └── tauri.conf.json
├── docs/
│   └── ARCHITECTURE.md      # Detailed architecture docs
├── Cargo.toml               # Workspace configuration
└── LICENSE                  # Apache 2.0 License

Running Tests

# Run all tests
cargo test

# Run specific test
cargo test -p frust-backend --test inference

Debugging

Enable debug logging:

RUST_LOG=debug cargo tauri dev

Performance

Benchmarks (Qwen 2.5 1.5B, Q4_K_M)

Hardware Prefill (tokens/s) Decode (tokens/s) VRAM Usage
NVIDIA GT 1030 ~45 ~25 ~720MB
NVIDIA RTX 3060 ~120 ~65 ~720MB
NVIDIA RTX 4090 ~350 ~180 ~720MB
Apple M1 Pro ~80 ~40 ~720MB

Memory Requirements

Model Size RAM Required VRAM Required Total
0.5B ~600MB ~400MB ~1GB
1.5B ~1.2GB ~720MB ~2GB
7B ~4.5GB ~2GB ~6.5GB

Troubleshooting

Common Issues

"No GPU found"

  • Ensure your GPU supports WebGPU
  • Update your GPU drivers
  • On Windows, enable "Graphics Features" in Chrome flags

"Out of memory"

  • Reduce max_seq_len in settings
  • Use a smaller model
  • Close other GPU-intensive applications

"Model download failed"

  • Check your internet connection
  • Verify HuggingFace Hub access
  • Try manual download to cache directory

Getting Help

  • Open an issue on GitHub Issues
  • Check existing issues for solutions

Contributing

We welcome contributions! Please see our guidelines:

  1. Fork the repository
  2. Create a feature branch (git checkout -b feature/amazing-feature)
  3. Commit your changes (git commit -m 'Add amazing feature')
  4. Push to the branch (git push origin feature/amazing-feature)
  5. Open a Pull Request

Code Style

  • Follow Rust standard formatting (cargo fmt)
  • Run clippy before committing (cargo clippy)
  • Add tests for new functionality

License

This project is licensed under the Apache License 2.0 - see the LICENSE file for details.

Acknowledgments

  • Candle - HuggingFace's ML framework for Rust
  • Tauri - Cross-platform desktop framework
  • Leptos - Reactive web framework for Rust
  • WebGPU - Next-generation GPU API

Made with ❤️ by the FRUST Contributors

About

No description, website, or topics provided.

Resources

Readme
Apache-2.0 license
Activity

Stars

3 stars

Watchers

2 watching

Forks

1 fork
Report repository

Releases

Packages

Contributors

Languages