proposal: Integrate Agent-Devtools for action step-tracing and visual inspection

[Jacopos311]

Feature Proposal: Step Tracing & Visual Inspection via Agent-Devtools

Summary

autoMate serves as a computer-using agent (CUA) executing local automation tasks based on user prompts. To make script execution, GUI interaction, and decision-making loops easier to monitor and debug, I propose adding support for Agent-Devtools tracing.

Problem / Motivation

Computer-use agents execute complex multi-step sequences (e.g., screen parsing, button clicks, system command execution):

• When an automation sequence fails or behaves unexpectedly, debugging requires sifting through terminal logs or custom print statements.
• Visually tracking the intermediate steps (prompt interpretation, tool/command execution, state transitions) in real-time is difficult during long automation runs.

Proposed Solution

Add an optional integration or event hook system for Agent-Devtools:

1. Action Hook System: Emit telemetry/trace events before and after each executed automation action (e.g., keypress, mouse event, tool call).
2. Opt-in Flag: Allow users to enable visual step tracing via an environment variable (e.g., AUTOMATE_DEVTOOLS=true) or a --trace CLI flag.

Benefits

• Real-time Debugging: Inspect action sequences step-by-step in an interactive dashboard.
• Improved Reliability: Easily pinpoint which step in a complex UI automation workflow failed or stalled.
• Better DX: Enhances developer experience when extending autoMate with new tools or system integrations.

Next Steps

What are the maintainers' thoughts on adding execution hooks for devtools/observability? I'd be glad to submit a draft PR with a lightweight PoC if this aligns with the project's goals.