Tiny LLaMA

Tiny-LLaMA

Pure-C# LLaMA inference engine. The hot-path kernels are written in C# and transcoded to CUDA by Hybridizer at build time, so the same source runs on the .NET thread pool or on the GPU depending on --backend managed|cuda. Loads any GGUF LLaMA model with F32, F16, Q8_0, or Q4_0 quantization; the recommended starter is TinyLlama-1.1B-Chat-v1.0 Q8_0 (~1.17 GB).

Source: src/7.AI/tiny-llama

# Default — managed (CPU) backend, stochastic decoding
dotnet run -c Release -- tinyllama-1.1b-chat-v1.0.Q8_0.gguf

# Deterministic CUDA fast path (CUDA Graphs)
dotnet run -c Release -- tinyllama-1.1b-chat-v1.0.Q8_0.gguf \
  --backend cuda --temperature 0 --top-p 1

The sample is bigger than the others in this repo — instead of one kernel, it's a full transformer (attention, RMSNorm, RoPE, SiLU/SwiGLU, Q8_0 matvec, softmax, sampler) plus a tokenizer and a GGUF reader. A few things specifically worth looking at:

• Same C# kernels on CPU and GPU. Math/FloatKernels.cs and Math/Q8Kernels.cs hold the kernels; Math/ParallelMath.cs is the managed dispatcher, Math/GpuBackend.cs is the CUDA dispatcher. Parallel.For becomes a CUDA grid loop on the device side and a thread-pool loop on the host side.
• CUDA Graphs fast path. When --backend cuda --temperature 0 --top-p 1, the per-token forward pass (~378 kernel launches) is captured once with cudaStreamBeginCapture and replayed via cudaGraphLaunch. Argmax stays on device; generated tokens accumulate in a device ring and are drained every 200 ms. See Utils/GraphInvoke.cs.
• CUDA satellite — extra translation units. The graph-capture path needs extern "C" host wrappers around cudaStream* / cudaGraph* APIs that Hybridizer.Runtime.CUDAImports does not expose. They live in intrinsics.cuh; host_wrappers.cu is a one-line #include that drags them into the nvcc build via <CudaExtraSources Include="host_wrappers.cu" /> in LlamaCsharp.csproj. The repo-shared CompileCUDA target in src/Directory.Build.targets splices @(CudaExtraSources) into the nvcc command line — empty by default for other samples. Without this wiring the DLL is missing llama_graph_* exports and the CUDA backend throws EntryPointNotFoundException at startup.
• Q8_0 matvec. The dominant cost in the forward pass. Math/Q8Kernels.cs keeps weights in the GGUF Q8_0 block layout (32 int8s + one fp16 scale per block) and dequantizes inline in the matvec, which keeps memory bandwidth — not arithmetic — as the bottleneck.
• GGUF loader. Gguf/GgufReader.cs parses the file format directly; no llama.cpp dependency. Tokenizer/BpeTokenizer.cs implements the SentencePiece-style BPE used by LLaMA models.
• Diagnostics env vars. LLAMA_DISABLE_GRAPH=1 keeps the non-default stream but skips graph capture/replay (A/B against per-call launches); LLAMA_KERNEL_PROFILE=1 forces cudaDeviceSynchronize after every launch and prints a ranked profile at exit.

Arguments

Position / flag	Type	Default	Description
(positional 1)	path	(required)	Path to a GGUF model file.
(positional 2)	string	"Once upon a time"	Prompt. Anything that doesn't start with --.
--max-tokens N	int	1280	Max tokens to generate (stops earlier on EOS or context limit).
--temperature T	float	0.7	Sampling temperature. 0 = greedy (argmax).
--top-p P	float	0.9	Top-p (nucleus) sampling. 1.0 disables the filter.
--backend NAME	enum	managed	Matvec backend: managed or cuda.

Tests

dotnet test -c Release

LlamaCsharp.Tests covers the tokenizer, the operators (deterministic forward pass vs. reference), and CUDA kernel conformance — useful as a worked example of testing transcoded kernels against their managed counterparts.

See the in-tree README for the full set of examples and the model-download links.