Is it possible to reduce GPU memory usage during inference? #19

Open

opened

on Nov 23, 2023

It appears it is taking >30GB of memory for inference. What are the parameters I can set which can reduce the demand?

Thanks!

to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

No one assigned

Labels

No labels

No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests