Feasibility of Having NanoLLM on non-Jetson? (GH200)

[DarkCTO]

We have a Grace Hopper 200 build and would like to have NanoLLM on it.
What would be the challenge and changes to be made to port NanoLLM over?

[dusty-nv]

@DarkCTO @johnnynunez has been building jetson-containers on GH200 recently, I think we would both agree vLLM/SGLang would be a better fit for taking advantage of the significant batching features and other high-throughput serving features those frameworks particularly excel at being intended for the larger GPUs -we are also tending to just use those for LLM/VLM inferencing on Jetson now with their chat.completion endpoints as they keep up with the model support and quantization methods. Does one of those or similar support the model(s) you are wanting? We do aim to build out more support for jetson-containers in Grace-based systems, but for NanoLLM the pace of model releases exceeded my ability to stay current with it (and fortunately those others and MLC got more mature with their VLM and chat.completion support)

I am interested to hear more about your use-case though and how we can otherwise help with these kind of edge2cloud containers depending on what you are into.