🤔 Copilot chat participant and tool behavior observations

[Dave Shoup (shouples)]

This post is going to serve as a living document / general feedback area about behavior we've seen when interacting with the @Confluent chat participant for Copilot (in "Ask" mode) and/or our extension's contributed tools.

We want notes and feedback here to help in a few areas:

• development to improve the experience with the chat participant and/or contributed language model tools (e.g. how can we better provide context for conversations?)
• collaboration with Microsoft with the growing chat/language APIs
• feedback for Microsoft and/or model vendors

Depending on the feedback, we may still file tasks/bugs in our Issues section.

[Matthew Seal (MSeal)]

Add interface function for LLM friendly rendering of results to be implemented. Hinting as to the type of response and content shape that is best known to be consumed would speed up debugging object->text rendering tests on copilot confusion. E.g. we added a layer for converting each of our objects to Markdown with some basic structures in place.

[Dave Shoup (shouples)]

Linking our src/chat/summarizers/ subdir for future reference: https://github.com/confluentinc/vscode/tree/e5ab9a880a029384c4a6595f40354c5ee9f2f4b1/src/chat/summarizers

[Dave Shoup (shouples)]

From the extension developer perspective, having a way to get more content/details out of these error 400 responses would be very helpful:

Error: Request Failed: 400 Bad Request
 {
    "errorType": "Error",
    "errorMessage": "Request Failed: 400 Bad Request\n",
    "errorStack":"Error: Request Failed: 400 Bad Request\n
        at v2._provideLanguageModelResponse (/Users/dshoup/.vscode/extensions/github.copilot-chat-0.27.0/dist/extension.js:478:7171)
        at processTicksAndRejections (node:internal/process/task_queues:95:5)",
}

The above was captured by our logError while handling the response stream, but appears to throw in here:

vscode/src/chat/participant.ts

Line 180 in 41cf98d

for await (const fragment of response.stream) {

Nothing else shown in the GitHub Copilot Chat output channel either:

[Dave Shoup (shouples)]

Discussion here: microsoft/vscode-discussions#2691

[Dave Shoup (shouples)]

We may need some guidance on how best to structure previous messages in the conversation, since https://github.com/confluentinc/vscode/blob/19df258fd754aedd501ab0cf9e499a964b5d710f/src/chat/summarizers/chatHistory.ts is trying to put them all in one markdown string with specific USER/ASSISTANT prefixes. Some models seem to lock in on that formatting:

[Lucia Cerchie (Cerchie)]

The tool invocation that is intended to be sent back to the model is of type LanguageModelToolResultPart where the content property is restricted to be an array of LanguageModelTextPart objects.
Now, since we aren't using LanguageModelPromptTsxPart, a tool's .invoke() is only ever returning LanguageModelTextPart instances in its LanguageModelToolResult, which are then LanguageModelToolResultPart constructor. That
LanguageModelToolResultPart will then be wrapped as a User message before being added to the message history. See original comment and usage here.

Before inspection of the type, a dev might wonder why LanguageModelToolResult would return only LanguageModelTextPart instances when not using LanguageModelPromptTsxPart, and not LanguageModelToolResultPart instances? In general, more documentation on these types and their relation to each other would be helpful.