请问可以在cpu上推理吗？大概要多少内存？谢谢

[wenli135]

如果把代码里所有对 cuda()的调用改掉，可以在cpu上推理吗？

[amith-ananthram]

@wenli135 I was able to do this!

[wenli135]

@wenli135 I was able to do this!

What changes had you made to the codes?

[amith-ananthram]

You can see my minimal edits here: main...amith-ananthram:Chinese-LLaVA:main

[wenli135]

You can see my minimal edits here: main...amith-ananthram:Chinese-LLaVA:main

thanks

[wenli135]

You can see my minimal edits here: main...amith-ananthram:Chinese-LLaVA:main

I use your code to run it on cpu, got error: TypeError: init() got an unexpected keyword argument 'attn_implementation',
then I remove argument attn_implementation="flash_attention_2" when initialize LlavaLlamaForCausalLM, but got another error: RuntimeError: "slow_conv2d_cpu" not implemented for 'Half'.

Any suggestion? thanks.

[sjh231]

You can see my minimal edits here: main...amith-ananthram:Chinese-LLaVA:main

I use your code to run it on cpu, got error: TypeError: init() got an unexpected keyword argument 'attn_implementation', then I remove argument attn_implementation="flash_attention_2" when initialize LlavaLlamaForCausalLM, but got another error: RuntimeError: "slow_conv2d_cpu" not implemented for 'Half'.

Any suggestion? thanks.

我绝对是Half表示半精度。然后可能cpu不支持半精度运算。因为我之前在使用BLIP2大模型的时候也是出现了这个问题，具体怎么解决我不太清楚，你可以问一下文心一言或者gpt？