Merge config directive #1012
simcop2387
started this conversation in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
I'm curious if you'd be onboard with someone sending a PR to add a
merge:parameter to the config that works using the --config-dir code for merging files. The reason I'd be wanting to do this is that I want to organize some things in my setup slightly different than just a single config-dir since i'm running multiple setups on different gpus and i'm wanting to setup some common files for a router i've got on top.The way i'm thinking of it working would be just as simple as this:
For my 2x P40 setup
and then for the A6000 48GB
And then I can leave out say the qwen36-27b for the A16 gpus (they're 16GB VRAM) so that they don't show up there.
That'd let me share the model definitions when appropriate, use macros for the little settings where the ampere gen stuff works but the P40 don't (like beellama kv-cache-tail on A6000 and A16 but not on the P40) without having to duplicate a bunch of stuff between everything, or having to have llama-swap show all models as being available everywhere even when not useful or even possible to load on some of them.
Otherwise i'm thinking i'll need/want to make a script that builds the final setup for each instance.
All reactions