Skip to content

E3: Hybrid cache (SSM/conv) + unified hybrid support gate #117

Description

@tlkahn

Hybrid cache (SSM/conv) + unified hybrid support gate

Cache and admit-path foundations for lfm2 / nemotron_h / qwen3_5_moe linear-attn layers.

Spec: doc/plans/engine-inference-core.md § E3

Part of: #106 (E3) · Epic: #56

Tasks - cache

  • SSM/conv state entries alongside KV entries (port SSMCacheEntry semantics)
  • Prefill chunk boundaries respecting SSM checkpointing (nextChunkEnd)
  • Create/reset/free on engine thread only

Tasks - support gate unification

Carried from PR #99 second review (finding 6), deferred until hybrid is first-class:

Today pure-dense qwen3_5 rejects hybrid via three overlapping checks:

  1. linear_num_key_heads > 0 || linear_num_value_heads > 0
  2. has_hybrid_layers (via shared reject_dense_common)
  3. Loop over layer_is_global when has_explicit_layer_types
  • Collapse into one shared hybrid gate helper
  • Family-specific allow-lists for accepted hybrid shapes
  • Remove pure-dense triple-reject once Stage E hybrid paths are real

Refs

Metadata

Metadata

Assignees

Labels

engine-inferenceEngine inference core (issue #51) feature areaenhancementNew feature or request

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions