Making fit evaluation of MoE models different

[geomi737]

Right now they are treated like standard dense models. But they can run really fast even with offloading. In fact thats the only comfortable way to launch big models with +-30t/s when there is not enough VRAM. So basically my idea is to eval their fit parameter by taking 1/4 of RAM in the system + VRAM. For example system with 16 gigs of ram and 8 gigs of vram can launch GPT-OSS 20B in Q4_K_M with MoE layers offloading at 25t/s. Its weight is 10 gigs. 4 gigs of RAM + 8 gigs is 12. So it fits and should be Good or Perfect.

And yes thats exactly my setup
llama serve -m /home/lesh/.cache/huggingface/hub/models--unsloth--gpt-oss-20b-GGUF/snapshots/d449b42d93e1c2c7bda5312f5c25c8fb91dfa9b4/gpt-oss-20b-Q4_K_M.gguf -ngl 999 -ncmoe 14 --no-mmap -fa on -ctk q8_0 -ctv q4_0 -np 1
thats offers me +-118k context window(or 72k i dont remember exactly, but should be 118k) with 25t/s