Had something quite identical on my todo list... Sharing what I think is missing

[johnr14]

Is your feature request related to a problem? Please describe.
The issue with agent is debugging and doing some small changes to the pipeline or prompts and wanting to see the change it will make.

Also versioning the agents and running automated tests to compare model performances with different prompts can result in a mess of files easily.

Describe the solution you'd like
A yaml history of a agent, that can be used for monitoring/debugging/experimenting.

Contains all initial variables passed to the agent, the list of agents executed with prompts (local copy of initial agent yaml), passed variables between agents, returned answer, token and time used, logic states of control flow nodes. Input agent file with sha256 hash and/or git commit for versioning. Can be used to generate graph (UML?) and view agent execution (real-time ?).

This is some sort of log and can be altered (changing response/prompt/agent order) and re-run, will only re-execute the agents needed and generate a new final answer.

orka run custom-workflow.yml -s custom-workflow_run_$(date +"%Y-%m-%d_%H-%M-%S").yaml
sed -i 's/second_prompt: YADA YADA/second_prompt: YIPI YIPI/g' custom-workflow.yml # Change the prompt 
orka run custom-workflow.yml -s custom-workflow_run_$(date +"%Y-%m-%d_%H-%M-%S").yaml
orka compare custom-workflow_run_2025-10-27_08-25-00.yaml  custom-workflow_run_2025-10-27_08-32-00.yaml # check differences between runs
sed -i 's/agent_3_response: SOME text/agent_3_response: FAKE BAD RESULT/g' custom-workflow_run_2025-10-27_08-32-00.yaml # second run looks better, but let's stimulate agent #3 failure
orka re-run custom-workflow_run_2025-10-27_08-32-00.yaml # let's see what impact agent #3 failure makes on the workflow.

-> check custom-workflow.yml using git/sha256
-> find differences, skip identical beginning (or use cached calls)
-> proceeding to agent_4 ...

Describe alternatives you've considered
Keeping a long list of agent_version_XXX.yaml to test various possibilities.

Additional context

• Not sure if you want to implement yaml import, to import a agent_some_feature.yaml into a workflow.yaml ? Could be used to reuse agents between workflow, or load a local-llm.yaml or default_model.yaml file.
• Also large prompts could be in separated files with .prompt to ease the reading of the workflow.yaml importing it with :

  prompt_file: path/my_prompt.prompt

• I have a wrapper on litellm to cache all requests/responses to ./llmcache using sha256 on the request as name, speeds up rerun of agent/workflow and a ttl to 24h, touch the cache file on each read to extent time to live. This really speed things up for troubleshooting ! You already have some caching mechanism, but caching outside the workflow would be easier than to implement it for multiple agents when debugging and that caching would not be use in production, your caching seems better for long term and more granular control.

• Reusing models seems repetitive, why not add a @model_name

# Fast responses (3-7B models)
- id: quick_response
  type: local_llm
  provider: ollama
  model: "llama3.2:3b"  # Smallest, fastest
  temperature: 0.7
  prompt: "{{ input }}"

would become :

- model_alias:
  name: llama3-local
  type: local_llm
  provider: ollama
  model: "llama3.2:3b"  # Smallest, fastest
  temperature: 0.7
- id: quick_response
  model_alias: llama3-local
  prompt: "{{ input }}"

---

That's about all I have to suggest. Will play with it some more. Thanks

[marcosomma]

@johnr14 Thanks for the thoughtful proposal. You nailed the real DX pain: tweak prompts, see impact, keep runs reproducible, and compare without a folder of brittle YAML variants.

I will work on it in to 0.9.6. I’ll post the minimal schema and tag you for review. Appreciate the thoughtful detail. I’ll keep you updated.👍🏼

[marcosomma]

This was move to 0.9.7 past release was focus on testing and reliability. Sorry about that!