Enhancement: An offline ChatGPT-like interface

[dnk8n]

Original title included, "fine-tuned to the content contained within Lokal". That is maybe a follow up step of lower priority.

An example - https://github.com/sumanentc/offline_chatgpt

[dnk8n]

Some useful docs: https://beebom.com/how-run-chatgpt-like-language-model-pc-offline/

[dnk8n]

This could also be relevant, at the moment it might not be feasible cost-wise to train such a model. And also, it is not expected to compete with the performance of chatGPT - https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm

Whatever we choose might soon be deprecated in favour of newer advances.

We should bear that all in mind.

[dnk8n]

There is also Alpaca from Stanford. Might need to train in the cloud, and distribute the model to GPU-less nimble devices

https://github.com/tatsu-lab/stanford_alpaca

[flemesre]

Would be interesting, but training and fine-tuning would require much more manpower and computational power than what's easily available at the moment. First example is really cool though - it's based on https://github.com/cocktailpeanut/dalai, which is a web app that can interact with C++ (CPU-optimised, good for Lokal x86 installs, ARM will be implementation-dependent) versions of either Stanford's Alpaca (already quantized, 7B parameter version is probably the most feasible with "only" 4GB of RAM and disk usage) or FB's LLaMa (much bigger if we don't use the quantized version). This would give any Lokal install that can spare the 4GB of RAM and CPU cycles a general-purpose LLM that can be used offline (!), even if it knows nothing about Lokal itself - there's still a lot of potential uses for that.

[katomaso]

What would you like it to train/fine-tune it for? AFAI understand it is a language model trained on instruction-based dataset of general English sentences. The first think that comes to mind is to re-train it for different languages. The question is if it isn't easier to just plug in a translation model.

[dnk8n]

Here is some insights into why you might want to fine tune, and also some alternatives to fine-tuning that are less resource intensive and more general purpose (using embeddings).

https://bdtechtalks.com/2023/05/01/customize-chatgpt-llm-embeddings/

But I agree, I don't think fine tuning or extra context is critical for the first step toward this issue.

[dnk8n]

Interesting, and on topic here - https://the-decoder.com/guanaco-is-a-chatgpt-competitor-trained-on-a-single-gpu-in-one-day/

[dnk8n]

Here is some more relevant information...

GPT4All: An ecosystem of open-source on-edge large language models.

GPT4All is an ecosystem to train and deploy powerful and customized large language models that run locally on consumer grade CPUs.