Skip to content

Local ONNX models with multi-shard external weights fail at ORT init; Harrier fp16 exports hit a CUDA GroupQueryAttention kernel gap #142

Description

@lemon07r

Found during the 2026-08-21 default-model screening (docs/benchmarks.md, "Default-Model Screening").

1. Multi-shard external weights unsupported. Vera's local-model config downloads a single ONNX external-data file (onnx_data_file). Models whose weights exceed one shard (convention: model.onnx_data, model.onnx_data_1, ...) fail at session init:

Exception during initialization: filesystem error: cannot get file size: No such file or directory [...]/onnx/model.onnx_data_1

Repro: lane {backend: custom-onnx, repo: onnx-community/harrier-oss-v1-0.6b-ONNX, onnx_file: onnx/model.onnx, onnx_data_file: onnx/model.onnx_data}. Workaround used for the screening: manual pinned download into a local dir lane.

2. fp16 GQA exports rejected by the CUDA EP. Harrier's model_fp16.onnx uses the fused GroupQueryAttention op with an attention bias, which the ORT CUDA kernel rejects:

Non-zero status code returned while running GroupQueryAttention node. Name:'/model/layers.0/attn/GroupQueryAttention' Status Message: attention_bias is not supported in GroupQueryAttention cuda kernel.

fp32 runs fine (the fusion is fp16-specific, so fp32 falls back to the unfused path). Today this only bites users who point VERA_LOCAL_EMBEDDING_ONNX_FILE at such an export; Vera's built-in GPU swap (quantized -> fp16) applies only to the jina preset, which exports a compatible graph.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions