Semantic search via embeddings & RAG support

[tredondo]

Would be nice to have the ability to perform vector searches across the subtitles in a channel, like with the abandoned yt-fts.

[h0lg]

Hey @tredondo, I wasn't aware of that project, but found these sections describing an LLM search feature.

Is that what you're requesting - the ability to have an LLM read through all your downloaded subtitles and generate summaries that you can full-text search and chat about with a bot?

[tredondo]

That's precisely it! In addition, instead of forcing OpenAI for generating embeddings, it would be nice to support the free https://docs.voyageai.com/docs/embeddings. As for vector DBs, there are a few that can be embedded / used offline (Chroma, Weaviate).

[h0lg]

I agree that semantic search capability would be nice.

AFAIU, the indexing would require the following steps:

1. Chunking subtitles (by token count, duration or topics)
2. generating LLM embeddings for chunks
3. store the chunks in a vector DB with metadata about the video, including time stamp

This alone should enable replacing or augmenting the full-text search results with semantic search results against the vector DB.

[h0lg]

On top of that, to enable "chatting about the subtitles" (retrieval-augmented generation RAG),
we'd have to wrap the search results in a system prompt and send it off to an LLM to answer the question.
@tredondo How would the LLM answer? Probably in natural language including links with timestamps instead of strictly structured search results (text at timestamp) like the full-text search?