Possible use case

[pors]

Not an issue, but I don't know where else I can submit this. I love this project, and I'm considering using it for my backend. I use traditional RAG for semantic search. To use structured RAG for the whole thing would become a bit costly, but a hybrid application could be very useful:

• take the top results of the semantic search and apply structured RAG only for this subset.
• this gives me explainability of the search results, and a better overall ranking quality of the top search results.

What do you think?

[gvanrossum-ms]

I think all the parts are somewhere under typeagent/knowpro (which contains the core algorithms). Search for KnowledgeExtractor.

I'd recommend caching the knowledge extracted by Structured RAG so that you incur the cost of knowledge extraction at most once per document.

I think you won't get the ultimate benefit of Structured RAG, because you are still limited by the ~60% recall that apparently plagues all vectorbases.

[gvanrossum-ms]

Unfortunately, our current code requires careful tuning of a variety of parameters (check out ConversationSettings) for each given model and each embedding model.

There are also the schemas to consider, and those definitely could use more tweaking:

typeagent/knowpro/answer_response_schema.py
typeagent/knowpro/answer_context_schema.py
typeagent/knowpro/search_query_schema.py
typeagent/knowpro/date_time_schema.py

[pors]

Yeah, I see a literal number in there :)
I'm afraid I can't avoid creating a labeled dataset to calibrate first. I'll let you know if I have any insights.

[gvanrossum]

If you want to, I can set you up with the data files (not in the repo) that we use to do evals. The dataset itself in in testdata/, but the JSON files giving expected outputs are not (for reasons I don't fully understand myself). The tools/query.py script has a --batch flag that sets it up for doing such evals.

[pors]

Yes, please. I'm very interested in that.

[gvanrossum]

Yes, please. I'm very interested in that.

These files are not for public consumption -- they don't represent a valid benchmark or anything, just a quick hack that we used internally. Therefore we don't want these files to go public and be interpreted as an official or valid benchmark or anything like that.

That said, if you email me at guido@python.org I'll mail you the files. (Others don't bother -- I need some evidence first that you're interested in helping us tuning the model.

[gvanrossum-ms]

(Also sorry for the late response. I need to tweak my notification settings...)