Awesome Local LLMs

A curated, category-organized registry of tools, models, and resources for running Large Language Models locally — on consumer GPUs, CPUs, Apple Silicon, and self-hosted hardware.

The goal is to be the definitive place to discover inference engines, quantized models, local chat UIs, model management utilities, fine-tuning tools, local RAG stacks, deployment options, and hardware acceleration layers that keep data on-device.

Inclusion criteria: Resources must be directly useful for running, serving, managing, or optimizing LLMs on local hardware. Cloud-only APIs and general tutorials are out of scope. See CONTRIBUTING.md for the full quality bar.

---

Contents

• Inference Engines
• Quantized Models
• Local Chat UIs
• Model Management
• Fine-Tuning Locally
• RAG Locally
• Deployment Tools
• Hardware / Acceleration
• Related Awesome Lists
• Contributing
• License

---

Inference Engines

Core libraries and servers that run LLMs on your own hardware.

• llama.cpp Community — High-performance C/C++ inference engine for GGUF models on CPU, CUDA, Metal, Vulkan, and ROCm.
  • Run: llama-cli -m model.gguf -p "Hello"
• Ollama Community — One-command runner and model server with an OpenAI-compatible API.
  • Run: ollama run llama3.2
• vLLM Community — High-throughput, memory-efficient serving engine with PagedAttention for local and on-prem deployment.
  • Run: vllm serve meta-llama/Llama-3.1-8B
• LocalAI Community — Self-hosted, OpenAI-compatible REST API for LLMs, embeddings, images, and audio.
  • Run: local-ai with a downloaded GGUF or safetensors model.
• ExLlamaV2 Community — Fast, memory-efficient inference for EXL2 and GPTQ quantized models on NVIDIA GPUs.
• TensorRT-LLM Official — NVIDIA-optimized inference library for low-latency, high-throughput LLM serving on RTX and data-center GPUs.
• MLX Official — Apple Silicon-native machine-learning framework with efficient LLM inference and training.
• llamafile Community — Single-file executable LLMs that run on macOS, Linux, and Windows without installation.
• KoboldCpp Community — Easy-to-use llama.cpp wrapper with a built-in web UI, story tools, and multi-backend GPU support.
• SGLang Community — Structured generation and fast serving runtime for local and distributed LLMs.

---

Quantized Models

Real model families and community quantization hubs that make large models runnable locally.

• Meta Llama 3 Official — Open-weight Llama 3 family from Meta, available in 8B and 70B sizes.
• Mistral 7B Official — Efficient open-weight model from Mistral AI, widely used for local inference.
• Qwen 2.5 Official — Alibaba's multilingual open-weight LLM family with strong local performance.
• DeepSeek-R1 Official — Reasoning-focused open-weight model family distilled into smaller local-friendly sizes.
• Gemma 3 Official — Google's lightweight, instruction-tuned open models in 1B, 4B, 12B, and 27B variants.
• Phi-4 Official — Microsoft's compact, high-quality open model well suited for consumer hardware.
• TheBloke GGUF Models Community — Massive collection of GGUF-quantized models for llama.cpp and Ollama.
• bartowski GGUF Models Community — Curated GGUF quantizations of popular open models.
• MLX Community Community — Ready-to-run MLX-converted models for Apple Silicon.
• Nous Hermes 3 Community — General-purpose, fine-tuned open model by Nous Research.

---

Local Chat UIs

Applications for chatting with local models through a friendly interface.

• Open WebUI Community — Extensible, self-hosted web interface for Ollama and OpenAI-compatible APIs.
  • Run: docker run -d -p 3000:8080 --gpus all -v ollama:/root/.ollama -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:main
• LM Studio Official — Desktop app to discover, download, and chat with local GGUF models.
• Jan Official — Open-source, privacy-first ChatGPT alternative that runs models locally.
• GPT4All Official — Offline chat application with a bundled model downloader and local-first defaults.
• AnythingLLM Community — Local and private workspace for chatting with documents and local LLMs.
• Lobe Chat Community — Modern chat UI supporting Ollama and other local providers.
• LibreChat Community — Multi-provider chat UI with Ollama and local endpoint support.
• Chatbox Community — Cross-platform desktop client for local and remote LLMs.
• Oterm Community — Terminal client for Ollama with conversation management.
• SillyTavern Community — Frontend for role-play and character-driven chats with local LLM backends.

---

Model Management

Tools for downloading, converting, quantizing, and organizing local models.

• Hugging Face Hub CLI Official — Official CLI and Python library to download and manage models from the Hub.
  • Install: pip install huggingface_hub
  • Download: huggingface-cli download TheBloke/Llama-2-7B-GGUF
• Ollama Model Library Community — Curated registry of models ready to pull and run with Ollama.
  • Pull: ollama pull llama3.2
• llama.cpp Convert Scripts Community — Scripts to convert Hugging Face checkpoints into GGUF format for local inference.
• AutoGPTQ Community — Easy-to-use GPTQ quantization toolkit for running compressed models locally.
• AutoAWQ Community — Quantization and inference library for AWQ-compressed models.
• ModelScope Official — Open model hub with CLI and SDK for downloading and managing models.
• Simon Willison's llm CLI Community — Command-line tool for running models via Ollama, OpenAI-compatible APIs, and local plugins.
  • Install: pip install llm
• Git LFS Official — Version-control extension used by Hugging Face and other hubs for large model files.

---

Fine-Tuning Locally

Frameworks for supervised fine-tuning, LoRA/QLoRA, and preference tuning on your own hardware.

• Axolotl Community — YAML-driven fine-tuning framework supporting LoRA, QLoRA, and full fine-tunes.
• Unsloth Official — Optimized fine-tuning and reinforcement-learning library with 2-5× speedups on consumer GPUs.
• LLaMA-Factory Community — Web UI and CLI for efficient fine-tuning of 100+ LLMs.
• torchtune Official — PyTorch-native library for post-training, fine-tuning, and quantization of LLMs.
• litgpt Community — Lightning-AI command-line tool for pre-training, fine-tuning, and deploying LLMs.
• mlx-lm Official — MLX examples for running and fine-tuning LLMs on Apple Silicon.
• PEFT Official — Hugging Face parameter-efficient fine-tuning library (LoRA, IA³, adapters).
• TRL Official — Hugging Face transformer reinforcement learning library for SFT, DPO, and PPO.
• bitsandbytes Community — 8-bit optimizers and 4-bit quantization primitives for local fine-tuning.

---

RAG Locally

Frameworks and applications for retrieval-augmented generation using local models and vector stores.

• AnythingLLM Community — All-in-one local workspace with document RAG, agents, and multi-model support.
• PrivateGPT Community — Local RAG and chat solution designed for privacy with no data leaving the machine.
• RAGFlow Community — Open-source RAG engine with document parsing and local model support.
• LangChain Official — Framework for building applications that connect local LLMs, vector stores, and tools.
• LlamaIndex Official — Data framework for ingestion, indexing, and querying with local LLMs.
• Haystack Official — End-to-end NLP framework for local RAG pipelines and semantic search.
• txtai Community — Lightweight local semantic search, RAG, and workflows in pure Python.
• Chroma Community — Developer-friendly open-source vector database for embeddings and RAG.
• Quivr Community — Second-brain RAG app with local model and vector-store support.

---

Deployment Tools

Servers, proxies, and packaging options for exposing local LLMs to clients.

• Docker Official — Container platform used to package local LLM servers with GPU passthrough.
  • Run Ollama: docker run -d --gpus all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama
• vLLM Community — Production-grade OpenAI-compatible server for local and on-prem deployment.
• LocalAI Community — Drop-in OpenAI-compatible API server for local models, embeddings, and media generation.
• llama.cpp Server Community — Built-in HTTP server in llama.cpp with chat-completion endpoints.
  • Build and run: make -C examples/server then ./server -m model.gguf
• TabbyAPI Community — OpenAI-compatible API server for ExLlamaV2 and llama.cpp backends.
• Text Generation Inference Official — Hugging Face production server for running open LLMs locally.
• OpenLLM Community — BentoML-based toolkit for serving open-source LLMs with an OpenAI-compatible API.
• exo Community — Run your own AI cluster at home across everyday devices (Mac, Linux, Windows, phones).
• distributed-llama Community — Connect home devices into a cluster to split LLM inference across nodes.
• llama-swap Community — Reliable model swapping proxy for any local OpenAI-compatible server.

---

Hardware / Acceleration

GPUs, SDKs, and backends that accelerate local LLM inference.

• NVIDIA CUDA Toolkit Official — SDK and runtime for GPU-accelerated inference on NVIDIA hardware.
• AMD ROCm Official — Open-source GPU compute stack for running LLMs on AMD Radeon and Instinct GPUs.
• Apple Metal Official — GPU framework powering MLX and llama.cpp on Apple Silicon.
• Intel OpenVINO Official — Toolkit for optimizing and accelerating inference on Intel CPUs, GPUs, and NPUs.
• ONNX Runtime Official — Cross-platform inference accelerator with support for CUDA, DirectML, ROCm, and OpenVINO.
• NVIDIA TensorRT Official — High-performance inference optimizer and runtime for NVIDIA GPUs.
• Intel Extension for PyTorch Official — PyTorch extension optimized for Intel CPUs and Xe GPUs.
• llama.cpp Vulkan Backend Community — Vulkan compute backend for cross-platform GPU inference on AMD, Intel, and NVIDIA.
• GGML Community — Tensor library powering llama.cpp backends for CPU, CUDA, Metal, Vulkan, and more.
• DirectML Official — Microsoft's DirectX-based machine-learning accelerator for Windows GPUs.

---

Related Awesome Lists

• vince-lam/awesome-local-llms — Curated list of local LLM tools and resources.
• ununununium/awesome-local-ai — 150+ open-source tools to run LLMs 100% locally.
• sam-blackfly/awesome-llm-tools — Essential tools spanning quantization, inference, agents, RAG, and fine-tuning.
• av/awesome-llm-services — LLM services, tools, and infrastructure for running AI locally.
• Hannibal046/Awesome-LLM — Broad awesome list of large language model resources.

---

Contributing

Read CONTRIBUTING.md for the quality bar, entry format, and PR process.

---

License

This list is released into the public domain under CC0-1.0.

Want us to build this for you?

Enterprise AI Atlas is maintained by Vibe Coding Agency. We prototype and ship agentic systems, MCP servers, and enterprise AI integrations for teams that need working software fast — without hiring a full AI engineering team.

Free guide: The Non-Technical Founder's Guide to Agentic AI — what agents and MCP servers are, and how to get a system built.