Start with the project README to build NInfer, download a published artifact, and run the CLI or HTTP server. Windows 11 users should also read the Windows guide.
| Document | Purpose |
|---|---|
| CLI | text, chat-history, image/video input, output streams, sampling, MTP, and common runtime options |
| HTTP serving | OpenAI Responses/Chat Completions, Anthropic Messages, state, streaming, token counting, authentication, and tool calls |
| Performance | RTX 5090 single-request and concurrent-decode results, MTP/DFlash measurements, and reproduction commands |
| Windows | native Windows 11 x64 requirements, vcpkg setup, build commands, and run notes |
| Perplexity | fixed-corpus and custom-text causal perplexity, comparison rules, progress, and reports |
| CLI examples | committed text, multimodal, thinking, long-decode, and long-context inputs |
The executable --help output is the exact source for command-line option spelling and defaults.
| Model | Weights | Download | Versioned model card source |
|---|---|---|---|
| Qwen3.6-27B | groupwise-int |
Hugging Face | model card |
| Qwen3.6-27B | nvfp4 |
Hugging Face | model card |
| Qwen3.8-27B | groupwise-int |
Hugging Face | model card |
| Qwen3.8-27B | nvfp4 |
Hugging Face | model card |
| Qwen3.6-35B-A3B | groupwise-int |
Hugging Face | model card |
The active references under maintainer/ record current architecture, model,
artifact, and maintenance contracts. These files are not additional user workflows or installed
API documentation.
The agreed model configuration, weight binding, and execution target architecture defines the intended model/artifact/Op boundaries, converter responsibilities, runtime support checks, and end-to-end design examples. It is a design contract, not a claim of implemented container or runtime support, and contains no migration plan. The references below continue to describe the delivered implementation.
Runtime and Op references:
- Engine architecture, execution ownership, scheduling, and request lifecycles
- Resource scheduling, continuation/checkpoint, and Device/Host context-cache contracts
- Paged KV context storage, ownership, and capacity model
- Operational logging channels, ownership, format, levels, and data policy
- Op admission, contracts, ownership, qualification, and performance rules
- ReplaySSM GDN technical reference
- Linear benchmark contract and registered suites
engine-architecture.md is the sole top-level Engine architecture reference.
resource-scheduling-and-context-cache.md is its narrower authority for resource selection,
materialization, checkpoint ownership, and replica policy. The remaining files define physical
storage, model, artifact, Op, or measurement contracts rather than parallel architecture variants.
Artifact and model references:
- NInfer artifact container
- Persistent tensor numeric formats
- Persistent storage layouts
- Qwen3.6-27B model semantics
- Qwen3.6-27B artifact contracts, including NVFP4
- Qwen3.8-27B DFlash2 mathematics and Engine state contract
- Qwen3.8-27B artifact contracts, including the NVFP4 target
- Qwen3.6-35B-A3B model semantics
- Qwen3.6-35B-A3B artifact contracts