inference-engineering

Learn LLM Inference Engineering step by step - from KV cache, PagedAttention, and continuous batching to vLLM, SGLang, and GPUs.

Rust + cuTile research prototype for paged latent-cache LLM decode attention, validated on an RTX 4060.

A practical handbook for software engineers to learn AI, Large Language Models (LLMs), and Inference Engineering—from fundamentals to production systems.

CPU-only conversational voice assistant with VAD-based speech segmentation, streaming LLM response generation overlapped with concurrent TTS synthesis via producer-consumer queues. Batch STT (Whisper). Fully instrumented with per-stage latency benchmarking.

LLM inference benchmarking dashboard: Python FastAPI backend with async orchestration, WebSocket live TTFT/TBT/throughput comparison across configs (512/128 to 4096/1024 tokens), Grafana + Docker Compose stack, GitHub Actions CI; 21/21 pytest passing.

it's me as a repository

Offline, evidence-first bottleneck hypotheses for LLM inference traces

Evidence-gated benchmark, comparison and promotion control plane for LLM inference serving changes.

Deterministic simulator for KV-cache admission, placement, movement, eviction, and recomputation policies

An interactive playground for learning inference engineering—explore LLM serving concepts, tune the stack, and graduate to production incidents.

Trace-Aware Serving Controller: eval-gated inference policy optimization

Interactive inference-engineering lessons and a bilingual AI mentor, built with Next.js and the Vercel AI SDK.

Executable field guide and deterministic labs for LLM inference engineering across kernels, scheduling, KV cache, placement, and control.

Deep Agents and SvelteKit harness for authoring verifier-gated Bonsai workflow packs.

TypeScript inference SDK for self-hosted LLM, ASR, TTS and embeddings. GPU lifecycle coordination, model handoffs and clients for vLLM, Whisper, Chatterbox, Qwen3 and Kokoro.