Quantization support needed for Ovis2.5 9B

[Dineshkumar-Anandan-ZS0367]

I have tried to quantize the model using ms-swift and this architecture is not supported yet. When can I expect or else share the quantization code for us.

[Dineshkumar-Anandan-ZS0367]

@Mihaiii @alibaba-oss @runninglsy @sshan-zhao

How to quantize the Ovis2.5-9B model (GPTQ-Int4)

[hfassold]

use the bitsandbytes 4-byte quantization instead

[wsbagnsv1]

Hey ill be uploading a sinq quant if you want tomorrow, can only be run with transformers and the sinq lib and not vllm yet but it works pretty well with my pr (;