Skip to content

[programming][machine_learning] llama.cpp performance tuning #1219

Description

@LukeShortCloud

For unified memory systems (ex., AMD Strix Halo, Macs, etc.), use these options to lower RAM usage at the cost of performance:

  --cache-ram 0 \
  --ctx-checkpoints 1 \

or increase performance at the cost of RAM:

  --cache-ram 1 \
  --ctx-checkpoints 0 \

https://www.reddit.com/r/LocalLLaMA/comments/1sq94qx/comment/oh7qlju/

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions