Hy3 support would be very welcome addition.
For background:
I can just about run MiniMax M2.7 at a low quant in 96GB (4x3090), and it's easily the most intelligent model I can run, likely due to ~64 layers adding a lot of nuance over something like Step or Qwen 122B. Notably, Qwen 3.6 27b is significantly better for its size vs. 3.5 122B, and one of the significant architectural differences is 48->64 layers.
Anyway, Hy3 has only 40B more params, but 80(!) layers. I'd expect that to be much better at complex coding tasks like refactoring and architecture, and the numbers seem to bear that out, showing performance much closer to M3 (a significantly larger model).
It would still be tight, REAPing would be required, but if exl3 can get a light reap down to ~2.75 bpw, then it's in play, for a lot of consumer 4x24GB or 3x32GB rigs.
Hy3 support would be very welcome addition.
For background:
I can just about run MiniMax M2.7 at a low quant in 96GB (4x3090), and it's easily the most intelligent model I can run, likely due to ~64 layers adding a lot of nuance over something like Step or Qwen 122B. Notably, Qwen 3.6 27b is significantly better for its size vs. 3.5 122B, and one of the significant architectural differences is 48->64 layers.
Anyway, Hy3 has only 40B more params, but 80(!) layers. I'd expect that to be much better at complex coding tasks like refactoring and architecture, and the numbers seem to bear that out, showing performance much closer to M3 (a significantly larger model).
It would still be tight, REAPing would be required, but if exl3 can get a light reap down to ~2.75 bpw, then it's in play, for a lot of consumer 4x24GB or 3x32GB rigs.