For unified memory systems (ex., AMD Strix Halo, Macs, etc.), use these options to lower RAM usage at the cost of performance:
--cache-ram 0 \
--ctx-checkpoints 1 \
or increase performance at the cost of RAM:
--cache-ram 1 \
--ctx-checkpoints 0 \
https://www.reddit.com/r/LocalLLaMA/comments/1sq94qx/comment/oh7qlju/
For unified memory systems (ex., AMD Strix Halo, Macs, etc.), use these options to lower RAM usage at the cost of performance:
or increase performance at the cost of RAM:
https://www.reddit.com/r/LocalLLaMA/comments/1sq94qx/comment/oh7qlju/