[Suggestion] Adopt a pattern to automate q/a when running skills in CI/CD, automated way

[cmoulliard]

Suggestion

We should adopt a pattern, or provide our own to automate the Q/A interactions when running the skills in CI/CD, automated way.
Such a pattern could be based on a YAML file detailing the response/answer, options, etc

Here is an example of such a file

# This file provides automated answers to common interactive prompts

intent_responses:
  - match_query: "Would you like to use the git workflow?"
    answer: "Yes"
    reasoning: "Authorized by project lead for full automation."

  - match_query: "Which migration strategy ?"
    answer: "Spring compat"
    options: ["Spring compat","Native Quarkus"]
    reasoning: "Selected 'Spring compat' to minimize breaking changes in the initial phase."

  - match_query: "Could you confirm the project is in the current workspace, or let me try to access it directly?"
    answer: "Yes"
    reasoning: "User is running the SKILLs within the application to be migrated"

default_behavior:
  ask_user_if_missing: false  # If a question isn't here, make the 'safest' choice
  log_decisions: true

WDYT ? @aureamunoz @maxandersen

[aureamunoz]

Yes, we should definitely invest in having a CI/CD mode for skills. However, there’s something about this approach that doesn’t quite seems right for me.
The core issue is that LLM output is non deterministic. The skill might ask "Would you like to use the git workflow?" today and "Should I create a dedicated branch?" tomorrow —> is the same meaning but different phrasing. String matching against natural language output is inherently fragile, and the YAML file would need constant maintenance to keep up with model variations.
The better approach would be to solve non-interactivity inside the skill itself, we could have a ## CI/CD Mode section that instructs the model to skip prompts and apply explicit defaults when CI=true. Anyway when launching a migration from the CI job, it should be in a non interactif mode, we can set the defaults in the skills itself.
I think what we need assertions on the results: Step 4 of the skill already has the six checks. In CI, those checks could be our assertions; if any of them fail, the pipeline fails. Then We could of course validate the output: application is compiling and starting up, tests passing, endpoints responding...