RAG on DuckDB Implementation Based on Prototype

[mulla028]

Description

Recently, we have implemented a prototype of RAG on DuckDB, and it proves that implementation is doable for the ChatCraft it's time to start working on it!

The implementation will take several steps, lets call them stages. Since we already have the set up of DuckDB using duckdb-wasm, the file loader, and format to text extractors, we are skipping some of the steps(stages). Therefore here are the steps we need to take in order successfully implement it:

Proposed Implementation Stages

1. Stage 1: Create Two New Tables in IndexedDB
   • Embeddings Table, with foreign key to a file
   • Chunks Table, with foreign key to a file
2. Stage 2: Implement Chunking Logic
   • Proper Chunking with overlap (cf. https://platform.openai.com/docs/assistants/tools/file-search#customizing-file-search-settings)
   • Proper Chunking Storage in IndexedDB
3. Stage 3: Implement Embeddings Generation
   • Allow using a cloud-based model or local (transformers.js or tensorflow.js)
4. Stage 4: Vector Search
   • Use DuckDB's extension Called VSS
   • Load Embeddings, Chunks, etc. into DuckDB
   • Apply HNSW Indexing to Increase Speed of the Search ( HNSW Indexing Provided by VSS extension)
5. Stage 5: LLM Integration
   • Modify Prompt Construction to Include Retrieved Context
   • Implement Source Attribution in Responses
   • Adjust Token Management to Account For Context
6. Stage 6: Query Processing
   • Implement Embedding Generation for User Queries
   • Use the Same Embedding Model as Documents for Consistency(text-embedding-3-small)

@humphd, @tarasglek please take a look at the proposed implementation stages, and approve them. Let me know if I am missing something :)

[humphd]

I made some edits. Basically, we should plan to store the chunks and embeddings beside the files in IndexedDB, so you can re-use them across chats (if they only live in DuckDB, they will get erased each time, which is wasteful).

We'll also need some way to indicate that we should do RAG vs. loading the entire file into the chat context. For example, when you attach files, we should probably decide based on the size of the file, and if it's over some threshold, we chunk and do embeddings as part of the import.

[mulla028]

Thank you for the correction!
Will follow edited proposal. My plan is to make a separate Pull-Requests for each stage. Will begin now, looking for your future feedbacks! Hopefully, we will land it smoothly ;D

[tarasglek]

i think it makes sense to start with bm25 search first rather than dive into embeddings. Will be easier and will provide a baseline for effectiveness of embedding based search

[mulla028]

It sounds interesting, never heard of it , have to make a research

[humphd]

@mulla028 he means this https://duckdb.org/docs/stable/extensions/full_text_search.html, where you would have DuckDB do a full-text search in the documents vs. creating chunks/embeddings beforehand.

@tarasglek he's already done the work of doing the embeddings/chunking, so I'd be OK if he continued with that approach, so he doesn't have to throw that work away.

[tarasglek]

Embedding approach is fine with me too

[humphd]

Thinking about this a bit more, I think instead of adding new tables, we should add new optional columns to the ChatCraftFile table:

https://github.com/tarasglek/chatcraft.org/blob/main/src/lib/db.ts#L87

We'd have:

• content - the binary data
• text - the extracted text, if any
• chunks - the chunked text, if any
• embeddings - the embedding arrays for the chunks, if any

[mulla028]

Sounds good !