Hi, I have a fork of this engine that currently runs Qwen3.6-35B-A3B in int4 ~2.5 tokens per second on an office laptop with Ryzen5 6600H and 16 GB RAM. I have an idea to train a model based on Qwen2-57B-A14B that would be adapted for disk streaming specifically. We could take that base model, add shared expert carrying ~40% of FFN mass, a pre-gated router trained so routing is computable at layer 1-2, and a strong MTP head, low top-k with trained threshold routing, Zipf-skewed expert utilization. We could probably train it with ~10000 dollars. I know it's funny, but if we get like 20 people who are interested and have not used their free trials on Google cloud, AWS, Azure, etc., we could just train sequentially and upload-download checkpoints to huggingface. And the model would just be MIT-licensed. If you are interested, leave a comment here, if you are willing to use paid resources (on vast or runpod) for this, you can also mention it, so I know how much computing power we have
Hi, I have a fork of this engine that currently runs Qwen3.6-35B-A3B in int4 ~2.5 tokens per second on an office laptop with Ryzen5 6600H and 16 GB RAM. I have an idea to train a model based on Qwen2-57B-A14B that would be adapted for disk streaming specifically. We could take that base model, add shared expert carrying ~40% of FFN mass, a pre-gated router trained so routing is computable at layer 1-2, and a strong MTP head, low top-k with trained threshold routing, Zipf-skewed expert utilization. We could probably train it with ~10000 dollars. I know it's funny, but if we get like 20 people who are interested and have not used their free trials on Google cloud, AWS, Azure, etc., we could just train sequentially and upload-download checkpoints to huggingface. And the model would just be MIT-licensed. If you are interested, leave a comment here, if you are willing to use paid resources (on vast or runpod) for this, you can also mention it, so I know how much computing power we have