It can be hard to keep track of whether changes to the system are improving things or not. Similarly, whether the usage of different LLMs results in noticeable changes. Implement a testing infrastructure that can hopefully be shared across bdcbot and dugbot.
- Use FAQ as initial test bed, add more example from ux team moving forward
- Store tabular format with questions, "correct" answer, and answers for different configurations, including LLM versions and code versions
- Could be useful both for testing and validation for papers
Related to #2
It can be hard to keep track of whether changes to the system are improving things or not. Similarly, whether the usage of different LLMs results in noticeable changes. Implement a testing infrastructure that can hopefully be shared across bdcbot and dugbot.
Related to #2