Skip to content

README.md

Latest commit

 

History

History
233 lines (188 loc) · 11.8 KB

README.md

File metadata and controls

233 lines (188 loc) · 11.8 KB
Raw

Tests

The retained tests protect current .ninfer, numerical operator, target, runtime-transaction, benchmark-report, and external protocol behavior. Repository verification principles are defined in ../AGENTS.md; Op contract and CUDA implementation guidance is in ../docs/maintainer/op-development.md.

Organization

  • artifact/ — Python container, registered layout, quantization, and resource behavior;
  • ops/ — one identifiable qualification suite per semantic Op or closely related overload group, using independent numerical/state-transition oracles at real supported shapes;
  • ops/linear/ — weight/activation-profile-specific public Linear conformance tests plus their one shared input generator, FP64 GEMM oracle, tolerance registry, and output/effects mechanics;
  • ops/linear_add/, ops/linear_pair/, ops/linear_swiglu/ — fused-Op suites split by registered weight/activation profile, each evaluating its complete formula rather than composing production Ops;
  • targets/qwen3_6/ — shared tokenizer/template, multimodal preprocessing, MRoPE, prepared-prompt, stop/output decoding, hybrid topology, decoder/GDN and round-state layouts/views, shifted-MTP alignment, Vision control, and family runtime mechanisms;
  • targets/qwen3_6_27b/ — registered inventory, converter recipe, source verifier, artifact bindings, reference diagnostics, family Program/multimodal/MTP behavior, and the opt-in real-Engine prefix test and causal-scoring State/KV isolation test;
  • targets/qwen3_6_35b_a3b/ — registered inventory/converter contracts, artifact-native diagnostic reference, MoE oracle, typed binding, selected-expert row access, 256K INT8 memory calculation, and the opt-in real public-Engine route;
  • test_ninfer_artifact_reader.cpp — C++ framing, directory, encoded-size, payload-span, and geometry behavior against a self-contained C++ fixture;
  • test_openai_schema.cpp, test_openai_responses.cpp, test_openai_responses_store.cpp, test_anthropic_schema.cpp, and test_tool_call_parser.cpp — current protocol translation, Responses Item/state/SSE behavior, schema-guided tool-argument normalization, structural fallback, and chunk-invariant incremental tool-call behavior;
  • test_request_log.cpp — the consumed request JSONL schema and exact measurement fields, plus representative Serve request/throughput pretty records, failure severity, zero-field elision, and exclusion of arbitrary client error text;
  • test_pretty_logging.cpp — observable Service/Tool prefixes and separation of executable identity from the human-readable record body;
  • test_http_error_handler.cpp — protocol-shaped payload-limit errors and application-error preservation;
  • test_ninfer_bench_support.cpp — product benchmark CLI, timing boundary, and schema-v14 reports;
  • test_bench_matrix.py — schema-v14 report consumption by the Python matrix summarizer;
  • test_serve_corpus.py — current serving request-log identity at the measurement consumer;
  • device/tensor/arena tests — reusable lower-component behavior; KV tests cover the core physical container, family runtime tests cover dimension-driven GDN storage/view mechanics, and Op tests cover mathematical state transitions at their own boundary.

Tests are grouped by observable risk, not by mirroring every source file or class. ops/op_tester.h and ops/op_check.h own only reusable device/guard and comparison mechanics. Concrete numerical criteria remain named by the semantic Op suite; there are no cross-Op tolerance presets.

ops/quantized_weight.h is the common packed-weight fixture for Q4/Q5/Q6/W8 and NVFP4 Op tests. It owns deterministic payload generation, device Weight views, row views, and independent logical weight decoding.

Build and run

cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=ON
cmake --build build --parallel
ctest --test-dir build --output-on-failure

Run a focused target for a localized change:

cmake --build build --parallel --target ninfer_sampling_test
ctest --test-dir build -R ninfer_sampling_test --output-on-failure

Enable uniform floating-point error records when establishing or reviewing an Op criterion:

NINFER_OP_REPORT_STATS=1 \
  ctest --test-dir build -V -R '^ninfer_(rmsnorm|softmax_attention)_test$'

Every participating comparison emits one OP_ERROR_STATS record containing the stable case label, actual error, active limit, and error-to-limit ratio. The switch changes reporting only; the same statistics still drive the normal verdict. Passing tests remain quiet without it.

The variable-width DFlash2 target-attention subset can be run with ./build/tests/ninfer_softmax_attention_test --dflash2-only. It covers D256/Q24/KV4 across all five cache codecs, W=2..16, B=1..8, request-local prefixes, cache effects, and Graph metadata/input updates. The default executable also runs the existing attention geometries and prefill tests.

Linear tests are independently runnable by weight and activation-compute profile:

cmake --build build --parallel --target \
  ninfer_linear_q4_a16_test ninfer_linear_q5_a16_test \
  ninfer_linear_q6_a16_test ninfer_linear_w8_a16_test
ctest --test-dir build -R '^ninfer_linear_(q4|q5|q6|w8)_a16_test$' --output-on-failure

All Linear files use ops/linear/linear_test_common.{h,cpp} and the same ops/quantized_weight.h fixture as the fused projection tests. The fixture produces the complete packed GPU payload and exact-decodes the logical float rows used by the one cpu_linear_gemm_fp64() reference. The reference performs naive double accumulation and never reproduces a production route's activation quantization, staging, reduction tree, or BF16 output rounding. Each activation compute path selects one centrally defined comparison tolerance for its whole suite; private kernel, schedule, launcher, and T selection do not change it. Individual test files call public linear() and contain no private selector, launcher, schedule, or kernel assertions.

Run the native Python suites with the project Python environment:

python3 -m pytest \
  tests/artifact tests/convert \
  tests/test_bench_matrix.py tests/test_serve_corpus.py

The Python suites cover generic artifact framing and exact converter inventories, source recipes, encoders, and payload verification. Model execution and real-artifact binding are tested through the C++ target and Engine suites below; there is no Python inference implementation. The two official source-resource preflight checks are opt-in: set NINFER_QWEN3_6_27B_MODEL and/or NINFER_QWEN3_6_35B_A3B_MODEL to the corresponding source checkpoint directory. Only those source-dependent checks are skipped when their variable is absent.

The C++ prefix/MTP integration test is separately opt-in because it loads the full artifact and runs the real engine:

NINFER_QWEN3_6_27B_WEIGHTS=$PWD/out/qwen3_6_27b.ninfer \
  ctest --test-dir build -R ninfer_qwen3_6_27b_prefix_real_test --output-on-failure

The causal-scoring integration test uses the same artifact variable and checks a full 1,024-column score tile, overlapping target suffixes, and repeated-window State/KV isolation:

NINFER_QWEN3_6_27B_WEIGHTS=$PWD/out/qwen3_8_27b_nvfp4.ninfer \
  ctest --test-dir build -R ninfer_qwen3_6_27b_score_real_test --output-on-failure

Run the peer 35B-A3B route independently:

NINFER_QWEN3_6_35B_A3B_WEIGHTS=$PWD/out/qwen3_6_35b_a3b.ninfer \
  ctest --test-dir build -R ninfer_qwen3_6_35b_a3b_real_test --output-on-failure

Without the corresponding variable CTest marks each C++ integration test as skipped. Neither test uses another numerical/execution path's generated tokens as a golden.

The capability-evaluation coordinator has its own environment and unittest entry point:

PYTHONPATH=eval eval/.venv/bin/python -m unittest discover \
  -s eval/tests -p 'test_*.py'

Run the serving contract manually after starting a resident server in another terminal:

./build/apps/ninfer-serve out/qwen3_6_27b.ninfer \
  --host 127.0.0.1 --port 18080
python3 -m tools.smoke.serve_contract \
  --base-url http://127.0.0.1:18080 --model qwen3.6-27b

This smoke check is intentionally not a CTest: it needs the real artifact, a supported GPU, and a server process that remains alive while the client exercises OpenAI Responses/Chat, Anthropic, state, streaming, and multimodal requests.

The thinking-preservation fixture starts and stops its own server, submits a fixed two-step tool history, compares restored and cold greedy output, compares stripped and preserved closed-turn prompt lengths, and verifies turn/response rewrite-checkpoint reuse paths plus Responses inheritance:

python3 tools/smoke/serve_thinking_preservation.py \
  --artifact out/qwen3_6_27b.ninfer --backend mtp

python3 tools/smoke/serve_thinking_preservation.py \
  --artifact out/qwen3_6_35b_a3b.ninfer --backend dflash

The shared messages are in fixtures/serve/qwen3_6_thinking_preservation.json.

What belongs here

A permanent test should protect one current risk, such as:

  • exact registered artifact bytes, geometry, object binding, or conversion transform;
  • a numerical operator contract with an independent oracle;
  • family Frontend or Program frontier, prefix, MTP, or multimodal behavior;
  • generated-token commit/stop/cancel consistency;
  • public benchmark or OpenAI/Anthropic observable behavior;
  • a reproduced supported bug.

Performance-only assertions belong in benchmarks and profiler review. Source scans, implementation-shape assertions, trivial getters/configuration, retired command surfaces, and broad additions without a concrete regression risk do not belong in the permanent suite.

DFlash2 Engine integration

The real test uses one explicit companion artifact and compares a fixed greedy fixture and its penalty-count variant with ordinary decoding. It also checks compact batches with unequal output budgets, same-seed stochastic replay, retained/fresh prefix behavior, and absence of a full backend KV pool. A shared DFlash/DFlash2 fixture starts decode at token 63, verifies across the page boundary, stops after one target column at token 64, and checks the exact retained frontier and subsequent generation with and without reuse. The KV Store test checks exact mapping and reservation accounting for the same transition. K>=7 also exercises a stop inside a licensed block; K=15 additionally checks oversized prefill, local ring wrap, and the logical context-capacity tail. Optional Vision runs image/video capture and prefix restore. Zero extra Device StateImage slots exercise Host snapshot/restore.

cmake --build build -j --target ninfer_qwen3_8_27b_dflash2_real_test
NINFER_QWEN3_8_27B_DFLASH2_WEIGHTS=out/qwen3_8_27b.ninfer \
  build/tests/ninfer_qwen3_8_27b_dflash2_real_test 15 1 1 8
NINFER_QWEN3_8_27B_DFLASH2_WEIGHTS=out/qwen3_8_27b.ninfer \
  build/tests/ninfer_qwen3_8_27b_dflash2_real_test 7 1 0 2 bf16 1 0
NINFER_QWEN3_8_27B_DFLASH2_WEIGHTS=out/qwen3_8_27b_nvfp4.ninfer \
  build/tests/ninfer_qwen3_8_27b_dflash2_real_test 2 0 0 2 int8

Arguments are K, Graph enabled, optimized head enabled, maximum B, target KV (bf16 or int8), Vision enabled, and extra Device StateImage slots. Defaults are 15 1 1 8 bf16 0 3. Run GPU integration tests serially. The individual Op suites remain the numerical/state-transition oracle; the fixed Engine fixture does not define bit parity across arbitrary floating-point routes.

The old/new Qwen3.8 binding matrix uses out/qwen3_8_27b_old.ninfer and out/qwen3_8_27b_nvfp4_old.ninfer for the legacy inventories, and the canonical filenames above for the companion artifacts. The legacy paths may be overridden with NINFER_QWEN3_8_27B_OLD_WEIGHTS and NINFER_QWEN3_8_27B_NVFP4_OLD_WEIGHTS.