Export and import of knowledge bases

[granludo]

The problem:

• Some kBs can be created more effectively on local high-powered computers where we can run crawlers, pdf parsers, whisper and other data extraction and sorting features.
• We might want to explore different chunking strategies or embeddings models for evals and such.

The solution:

• export all the files related to a knowledge base and bundle them into a zip file , with a json manifesto of all the files and how it needs to be ingested
• develop a import kb ingestion plugin (it might make calls to other existing plugins to get the job done)

this way a user can create his kbs on a local version of lamb-kb-server ... export-them and push them into the hosted version of lamb.