Doesn't work with 4bit?

[dreemur99]

Tried using with several ggml bins (some from TheBloke, some from GPT4All) and it seems like this code only works with models not labeled as 4bit? Or am I missing something?

[Macoron]

You're right, it doesn't work right now. Current compiled libraries of gpt4all are too old and doesn't support modern architecture or quantization. Rebuilding them should fix that. Check #11 for more context.

[dreemur99]

I tried rebuilding but it doesn't seem to work? (again) Unless I am missing something. Rebuilt and replaced the files but now none of the llms load.

[Macoron]

I tried rebuilding but it doesn't seem to work? (again) Unless I am missing something. Rebuilt and replaced the files but now none of the llms load.

Yes, they also changed C API, so current C# bindings should also be rewritten. Also they introduced some dynamic dll loading, so you would need to handle that too.
https://github.com/nomic-ai/gpt4all/blob/main/gpt4all-backend/llmodel_c.h