Skip to content

Work out the context limit the way vLLM does #8

Closed

Description

@layer8labs8

The problem

QPF016 and the setup at the end of each report take a model's context limit from max_position_embeddings alone. vLLM works it out in _get_and_verify_max_len and derive_max_model_len_and_key, and three differences matter:

  1. It reads other keys too, such as n_positions, max_seq_len, and seq_length, takes the smallest, and lets model_max_length in the model config override them.
  2. It takes the smaller of that and model_max_length in tokenizer_config.json.
  3. It multiplies by the RoPE scaling factor, except for the su, longrope, llama3, yarn, deepseek_yarn, and deepseek_llama_scaling types, and except for Gemma 3. A linear factor of 4 on a 4,096-token model allows 16,384 tokens, so today QPF016 raises a false alarm at 8,192.

What to do

  1. Add a helper in src/quantpreflight/rules.py that returns the limit from the config and the tokenizer config, following the three steps above. The checkpoint's tokenizer config is in meta.tokenizer_config.
  2. Use it in _check_qpf016_context_limit and in plan_setup in src/quantpreflight/planner.py.
  3. Add a test for each step, including one where the RoPE type is yarn and the factor must not apply.

Test your change

pip install -e ".[dev]"
pytest

New to the project? See CONTRIBUTING.md. Comment here if you get stuck.

Activity

  1. mikemikimike commented on Nov 6, 2026

    @mikemikimike
    Contributor

    Implemented in PR #19: #19

    The fix derives the effective vLLM context limit from the supported model config keys, model and tokenizer limits, and RoPE scaling rules, then uses the same value in QPF016 and setup planning. Tests cover configuration precedence, tokenizer caps, linear scaling, and the scaling exceptions.

    Validation:

    • python -m pytest -q (7556 passed)
    • python -m pytest tests/test_rule_boundaries.py tests/test_planner.py -q (238 passed)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't workinghelp wantedExtra attention is needed

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions