Multiple configuration profiles for the same gguf file

[ovidiusoft]

This might be a stupid question: if I define multiple configurations (models) over the same gguf file, will that file be completely reloaded from disk on model "change"? For example, Unsloth's docs for Qwen 3.5 ( https://unsloth.ai/docs/models/qwen3.5#recommended-settings ) would suggest 4 different "models", depending on the usecase, but they would point to the same gguf file on disk. What would be the best way to configure llama-swap so all "models" exist, while minimizing model reload (this is also useful for multi-agent setups like OpenClaw, if an agent's model is unloaded fun things could happen).

Thank you!

[ovidiusoft]

Looks like I am not the only one running into this situation, if this idea catches, we might get support in llama.cpp / llama-server - ggml-org/llama.cpp#20113

[mostlygeek]

Have you seen the setParamsByID filter configuration?

For example:

# show aliases in v1/models
includeAliasesInList: true

models:
  "Q3.5-35B":
    env:
      - "CUDA_VISIBLE_DEVICES=GPU-6f0,GPU-f10"
    filters:
      stripParams: "temperature, top_k, top_p, repeat_penalty, min_p, presence_penalty"

      # new filter
      setParamsByID:
        "${MODEL_ID}:thinking-coding":
          temperature: 0.6
          presence_penalty: 0.0
        "${MODEL_ID}:instruct":
          chat_template_kwargs:
            enable_thinking: false
          temperature: 0.7
          top_p: 0.8

    cmd: |
      ${server-latest}
      --model /path/to/models/Qwen3.5-35B-A3B-UD-Q6_K_XL.gguf
      --ctx-size 262144
      --fit off
      --temp 1.0 --min-p 0.0 --top-k 20 --top-p 0.95
      --repeat_penalty 1.0 --presence_penalty 1.5

[ovidiusoft]

I have not! 😱 This looks to do exactly what I wanted, have one base model and a few variations. Thank you so much, I will test it asap!

[shorthopdigital]

Hey, piggybacking here since I think I am all turned around.

Any idea why this config keeps throwing "Error loading config: yaml: line 14: did not find expected key"?

logLevel: info
logToStdout: "both"
startPort: 10001
includeAliasesInList: true

models:
  qwen3.5-9b-reasoning:
    ttl: 120
    env:
      - "HSA_OVERRIDE_GFX_VERSION: 10.3.1"
      - "AMD_VULKAN_ICD: RADV"
      - "MODEL_ID: qwen3.5-9b-reasoning"
    name: "qwen3.5-9b-reasoning"
    filters:
      stripParams: ""
      setParamsByID:
        "qwen3.5-9b-reasoning:vision"
          mmproj: "/models/Qwen/mmproj-F16.gguf"
    cmd: >
      llama-server
      --host 0.0.0.0
      --port ${PORT}
      --model /models/Qwen/Qwen3.5-9B-UD_Q5_K_XL/Qwen3.5-9B-UD_Q5_K_XL.gguf
      --alias qwen3.5-9b-reasoning-32k
      --jinja
      --verbose
      --chat-template-kwargs '{"enable_thinking":true}'
      --ctx-size 32768
      --n-gpu-layers 999
      --batch-size 512
      --ubatch-size 512

If I remove/comment out the "filters" line, it works fine?

[shorthopdigital]

Hey, piggybacking here since I think I am all turned around.

Any idea why this config keeps throwing "Error loading config: yaml: line 14: did not find expected key"?

logLevel: info
logToStdout: "both"
startPort: 10001
includeAliasesInList: true

models:
  qwen3.5-9b-reasoning:
    ttl: 120
    env:
      - "HSA_OVERRIDE_GFX_VERSION: 10.3.1"
      - "AMD_VULKAN_ICD: RADV"
      - "MODEL_ID: qwen3.5-9b-reasoning"
    name: "qwen3.5-9b-reasoning"
    filters:
      stripParams: ""
      setParamsByID:
        "qwen3.5-9b-reasoning:vision"
          mmproj: "/models/Qwen/mmproj-F16.gguf"
    cmd: >
      llama-server
      --host 0.0.0.0
      --port ${PORT}
      --model /models/Qwen/Qwen3.5-9B-UD_Q5_K_XL/Qwen3.5-9B-UD_Q5_K_XL.gguf
      --alias qwen3.5-9b-reasoning-32k
      --jinja
      --verbose
      --chat-template-kwargs '{"enable_thinking":true}'
      --ctx-size 32768
      --n-gpu-layers 999
      --batch-size 512
      --ubatch-size 512

If I remove/comment out the "filters" line, it works fine?

Oof, nevermind. I am fat fingering all over the place today. Fixed like 6 typos above and all is working (most relevant, the missing colon on the line in question). Carry on!

[ovidiusoft]

There must be something I'm not doing right :( . This is my config file. Loading models works fine, groups work fine, but I don't see the aliases in the UI or on /v1/models... 🤷‍♂️ I am running the v197-cuda-b8265 docker image (and any other models in normal configuration work just fine).

config.yaml

[ovidiusoft]

I might have hit on this issue... 🤔 - #576

[mostlygeek]

Not sure if you solved it yet.

The issue with your config is that setParamsByID is a filter so it should be under the filters key. I fixed it up for you. Also llama-server will use the defaults you define on the CLI so it is unnecessary to set them again in setParamsByID. I removed those and it shortened things by a bit.

healthCheckTimeout: 180
includeAliasesInList: true
logLevel: "debug"
logToStdout: "both"
sendLoadingState: true

macros:
    "latest-llama": >
        /app/llama-server
        --port ${PORT}

groups:
    "chat-models":
        swap: true
        exclusive: false
        members:
            - "Qwen3.5-35B-A3B"
    "embedding-models":
        swap: true
        exclusive: false
        members:
            - "Qwen3-Embedding-8B"

models:
    #####################################################
    # QWEN 3.5
    ####################################################

    "Qwen3.5-35B-A3B":
        name: "Qwen3.5-35B-A3B"

        env:
            - "MODEL_ID=Qwen3.5-35B-A3B"

        filters:
            stripParams: "temperature, top_k, top_p, repeat_penalty, min_p, presence_penalty"
            
            # Make sure setParamsByID lives under the "filters" key
            setParamsByID:
                "${MODEL_ID}:coding":
                    temperature: 0.6
                    presence_penalty: 0.0

                "${MODEL_ID}:instruct-nothink":
                    temperature: 0.7
                    top_p: 0.8
                    chat_template_kwargs:
                        enable_thinking: false

                "${MODEL_ID}:reasoning-nothink":
                    chat_template_kwargs:
                        enable_thinking: false

        cmd: |
             ${latest-llama} --model /models/unsloth/Qwen3.5-35B-A3B/Qwen3.5-35B-A3B-UD-Q4_K_XL.gguf --mmproj /models/unsloth/Qwen3.5-35B-A3B/mmproj-F16.gguf
             --ctx-size 262144
             --seed 3407
             --temp 1.0
             --top-p 0.95
             --top-k 20
             --min-p 0.00
             --presence_penalty 1.5
             --repeat_penalty 1.0
             --jinja
             --chat-template-file /models/unsloth/Qwen3.5-35B-A3B/chat_template.jinja


    ###################################################
    # EMBEDDING MODELS
    ###################################################

    "Qwen3-Embedding-8B":
        cmd: ${latest-llama} --model /models/qwen/Qwen3-Embedding-8B-Q4_K_M.gguf -ngl 0 --device none --embeddings

[ovidiusoft]

Oh wow, death by yaml indentation 🤦‍♂️. Thank you so much! Also, thank you for the smaller version, I did not realise those arguments were redundant. I tested now and all works as expected! <3

[superjamie]

Following this method you posted on Reddit the model list ends up with three models, the base model and the two filter variants:

• Q3.5-35B
• Q3.5-35B:thinking
• Q3.5-35B:instruct

This is a bit messy. Is it possible to filter parameters on the base model name, so you'd end up with just a base model and thinking variant in the model list?

I mean a config like this. Notice there is no ${MODEL_ID}:instruct filter, there is only ${MODEL_ID} and ${MODEL_ID}:thinking:

models:
  "Q3.5-35B":
    filters:
      stripParams: "temperature, top_k, top_p, repeat_penalty, min_p, presence_penalty"
      setParamsByID:
        "${MODEL_ID}":
          chat_template_kwargs:
            enable_thinking: false
          temperature: 0.7
        "${MODEL_ID}:thinking":
          chat_template_kwargs:
            enable_thinking: true
          temperature: 0.6
    cmd: |
      ${server-latest}
      --model /path/to/models/Qwen3.5-35B-A3B-UD-Q6_K_XL.gguf

(full parameters omitted for brevity, the model documentation has the correct parameters)

[mostlygeek]

Hi,

Yes, you can use just the ${MODEL_ID} macro as the key and it'll match correctly.

The keys in setParamsByID are just strings that are checked to see if they match the model in the request. They can be anything really. For example, you can use "abc", "m1" or "q::thinker" as they would be valid keys. Aliases will automatically be created for them. I prefer to use the ${MODEL_ID} macro because I like the style but there's specific requirement for it.

[superjamie]

Awesome, thank you

[superjamie]

Is there a way to support GLM's thinking format? According to the documentation it is changed with:

"thinking": {
    "type": "disabled"
}

I tried a few different syntaxes but could not get it to disable thinking.

[ovidiusoft]

What chat template are you using? Looking at the Unsloth one here https://huggingface.co/unsloth/GLM-4.7-Flash-GGUF?chat_template=default, it looks like the variable is called "enable_thinking"?

[superjamie]

That's it, thank you!

  "zai-org/GLM-4.7-Flash":
    filters:
      setParamsByID:
        "${MODEL_ID}":
          chat_template_kwargs:
            enable_thinking: false
        "${MODEL_ID}:thinking":
          chat_template_kwargs:
            enable_thinking: true

[mpetruc]

Hi, sorry to resurrect this thread, but i have a very hard time understanding how to use aliases with seParamsByID.
I configured several aliases for a model using setParamsById. However, in the dashboard there is only one entry (for the base config) with the aliases listed on the next line in smaller, whiter fonts, not clickable. There is only one Load button for the whole group of aliases, which loads only the base configuration. Api calls using model:alias as model name also load only the base config.
i thought that my config was messed up, so i wiped it and just copied and pasted @mostlygeek example above, just to see if it produces different results. Turns out that it doesn't. Here's how the model appears in my dashboard:

So, i don't think there's anything wrong with my own config. But how can I load just the specific alias i need ?!? Thank you for your help and your patience!

[mostlygeek]

Hi,

Aliases only work when going through the API endpoints.

If you're going through upstream/ and using the llama-server UI the filters don't affect that traffic.

I've been considering adding a new config to enable filtering into the upstream/ traffic.

[superjamie]

Have you missed this additional required global setting:

includeAliasesInList: true

As the name suggests, that is needed for the aliases to be included in the /models list.

I made this same mistake myself when I first tried it.

[mpetruc]

no i din't miss includeAliasesInList: true - . As i mentioned, the screenshot is from the dashboard of a llama-swap instance using the @mostlygeek config from the message "Marked as Answer".

I also tried the API endpoinds. As i mentioned above: "Api calls using model:alias as model name also load only the base config".
Are there any limitations in terms of flags that can be included in the setParamsByID? E.g. mmproj, sm, fit-margin, b?

healthCheckTimeout: 30000
logLevel: debug
metricsMaxInMemory: 1000
includeAliasesInList: true

macros:
  "ik_llama_server": >
    /home/user1/llama.cpp/build/bin/llama-server
    --port 5050
    --host 0.0.0.0
  ik_proxy: http://127.0.0.1:5050
   models: /models

models:
  "qwen3.5-122b-Q5-ik":
    proxy: ${ik_proxy}
    cmd: |
      ${ik_llama_server}  
      -m ${models}/Qwen3.5-122B-A10B-smol-IQ5_KS.gguf  -c 65535 --fit --temp 1.0  --repeat-penalty 1.0 --presence_penalty 0.0   --parallel 1      
    filters:  
      setParamsByID:  
        "qwen3.5-122b-Q5-ik:vision":  
          mmproj: ${models}/AesSedai-Qwen3.5-122B-A10B-GGUF/mmproj-Qwen3.5-122B-A10B-F16.gguf 
          fit-margin: 2048  
          sm: layer
          ub: 512 
          b: 1024  
        "qwen3.5-122b-Q5-ik:text":  
          fit-margin: 256  
          sm: graph
          no-mmap: 
          ub: 2048 
          b: 2048
          ```

 In the meantime i just created two separate models, "qwen3.5-122b-Q5-ik-vision and "qwen3.5-122b-Q5-ik-text.
thank you,

[mostlygeek]

setParamsByID injects new JSON values into the request based on the model name before it hits the upstream. For llama-server this can dynamically control temperature, top-p, top-k, chat template kwargs, etc at request time.

For things that require restarting llama-server, like setting mmproj, you will require a new model configured.

[mpetruc]

aaah, makes sense now. Thank you!

[chrislawso]

First wanted to say thank you to devs for excellent work!

I found these features and discussion and will like to test if can achieve goal.

Use case is an external web application which only supports specifying a single custom base URL for openai api, and only can specify a single model name. In other words the application has no internal mechanism for routing certain prompts to different endpoint or different port or different model alias names.

The goal to circumvent the above constraints was to try to use llama-swap for multiple configuration profiles for the same gguf file ie route different chat requests to same gguf but some to enable_thinking: false and others to enable_thinking: true. I want certain prompts to process using disabled thinking reasoning for faster output response, other certain prompts can be processed with thinking reasoning enabled. Again, we cannot change the base url or the model name in the external application config. Also to state the obvious, the inference engine machine is hardware constrained, has no additional pcie slots and cannot install different or larger vram gpu, cannot run additional llamacpp of same duplicate gguf redundantly to simply serve different parameters.

Assuming the external application constraints, can llama-swap filter or route chat requests based on regex or certain text ie some of the prompt requests allows to customize the "role": "system" system prompts "content", while other areas of external application does not allow custom system prompt content, instead can only customize the user content. We could insert into system or user content ie immediate beginning line enable_thinking: false or enable_thinking: true which llama-swap could then pre-process and then llama-swap internally modifies the model:alias name or applies a different associated macro filter rule.

What if any, are the other macro's that llama-swap supports for filtering besides ${MODEL_ID} ? For example, since we cannot specify different parameters for ${MODEL_ID} , are there any macro filter parameters similar to ${SYSTEM_PREFIX_ID} or any other recommendations for how to match beginning part or the whole a particular system prompt "content" ?

Does llama-swap support performing routing logic based on system prompt "content"?

How could llama-swap be configured to perform routing to different setParamsByID based on certain preset system prompt content ?

If not currently possible, can I open a feature request if this is a reasonable and possible new feature for llama-swap?

Thanks

[mostlygeek]

Hi,

llama-swap doesn't support the level of rewriting or filtering as your described. What may be better than supporting it in llama-swap is to create a simple go proxy binary so the request path is:

client --> (rewrite-proxy) --> llama-swap --> inference server

rewrite-proxy can capture the logic you have above and rewrite the model name in combination with llama-swap's setParamsByID to enable/disable thinking.

It would be a quick job with GLM5.2 or any of the smarter models.