Skip to content

Commit 4b543a7

Browse files
committed
feat(studio): add multidimensional comparison debugging
1 parent 1bab63b commit 4b543a7

47 files changed

Lines changed: 9130 additions & 2060 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

frontend/README.md

Lines changed: 46 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -114,15 +114,27 @@ server that `veadk frontend` launches — no separate backend.
114114
- **Custom-agent workbench**: configure an agent with a rich Markdown
115115
system-prompt editor (including heading and list shortcuts), then debug with
116116
expandable, copyable runner error details, per-result Trace inspection, and
117-
review. In-progress drafts are stored only in the current browser and scoped
117+
review. Debugging is optional: creators can enter comparison debugging or
118+
publish directly after architecture configuration. Comparison results keep
119+
the baseline and candidates side by side, while one focused configuration
120+
panel preserves edits when switching Agent or dimension. The baseline exposes
121+
a read-only model, prompt, and Skills summary; candidate model changes support
122+
Model ID, Provider, API Base, and temporary API Key values. In-progress drafts
123+
are stored only in the current browser and scoped
118124
to the signed-in user. MCP tokens are converted to Runtime environment
119125
variables: generated source retains only the `${ENV_NAME}` reference, while
120126
YAML and browser drafts preserve the corresponding environment value.
121127
Runtime updates reload existing values, and the deployment form keeps all
122128
environment values visible to users who can view the Agent. Entering a
123129
replacement Token overrides the previous value. Long descriptions and prompts
124130
scroll within bounded editors, while the sidebar stays pinned to the
125-
viewport. On narrow desktop windows, the structure, configuration, and debug
131+
viewport. Changing the group set or any candidate model, prompt, credential,
132+
or Skill configuration makes the previous comparison Session read-only.
133+
Studio keeps the old transcript visible until the user confirms a new Session,
134+
then starts all valid groups with one new comparison ID without replaying prior
135+
inputs. If any group fails to start, Studio cleans the newly staged runs and
136+
preserves the previous evidence. On narrow desktop windows, the structure,
137+
configuration, and debug
126138
panels stack vertically instead of squeezing the form. The deployment page
127139
pairs an inspectable Agent topology with a vertically aligned action rail for
128140
YAML export, source download, and the code browser/editor dialog, while keeping
@@ -432,6 +444,38 @@ Deleting a draft attachment deletes its object. Deleting a session deletes all
432444
media scoped to that session from either backend. Because `/tmp` may be cleared
433445
at any time, use TOS when attachments must survive process or host replacement.
434446

447+
## Multi-scheme debug comparison
448+
449+
The Custom Agent editor can compare its read-only baseline with up to three
450+
candidate schemes in one debug workspace. A candidate may change multiple
451+
Agents and dimensions together. The supported dimensions are system prompt,
452+
model, and skills. Model changes treat Model ID, provider, and API Base as one
453+
atomic configuration.
454+
455+
Model API keys are temporary debug credentials. A key entered while creating a
456+
Custom Agent may be left blank so the Studio server resolves its configured
457+
credential. A candidate with a custom API Base must provide a temporary key.
458+
The browser keeps these keys outside the persisted draft and comparison
459+
history; the server injects them only into the selected debug environment.
460+
Applying a candidate copies configuration and in-memory credential state, but
461+
never writes a key into generated source, the Draft, or local storage.
462+
463+
All running schemes receive the same initial text input. Interactive A2UI
464+
actions are broadcast only when every running scheme exposes the same action;
465+
otherwise the affected runs are marked as input-diverged. Changing any scheme
466+
makes the previous Session read-only. Starting a new Session requires explicit
467+
confirmation, starts all schemes atomically, and never replays prior inputs. If
468+
any environment fails to start, Studio cleans the newly staged runs and keeps
469+
the previous Session evidence available.
470+
471+
The workspace reports first visible text latency, total latency, completed tool
472+
calls, and total tokens. Cost is shown as unavailable until a shared accounting
473+
contract exists. Trace alignment uses exact invocation or tool-call identifiers
474+
and leaves unmatched events explicit. Human verdicts and reasons are stored in
475+
the local comparison record; applying a candidate requires an explicit
476+
`Adopt candidate` verdict. Scenario evaluation remains disabled until its API is
477+
available.
478+
435479
## Skills and sub-agents
436480

437481
Type `/` in the composer to search skills mounted on the selected agent. Type

frontend/src/a2ui/Surface.tsx

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -79,12 +79,18 @@ function Fallback({ node }: { node: A2uiComponent }) {
7979
export interface SurfaceViewProps {
8080
surface: SurfaceState;
8181
onAction: (action: A2uiAction | undefined, node: A2uiComponent) => void;
82+
readOnly?: boolean;
8283
}
8384

8485
/** Render a single surface starting at its root component. */
85-
export function SurfaceView({ surface, onAction }: SurfaceViewProps) {
86+
export function SurfaceView({
87+
surface,
88+
onAction,
89+
readOnly = false,
90+
}: SurfaceViewProps) {
8691
const ctx: RenderContext = {
8792
surface,
93+
readOnly,
8894
resolve: (v: DynamicValue) => resolveValue(v, surface.dataModel),
8995
resolveString: (v: DynamicValue) => resolveStr(v, surface.dataModel),
9096
dispatchAction: onAction,

frontend/src/a2ui/components/Button/Button.tsx

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ export function Button({ node, ctx }: ComponentRendererProps) {
99
className={`a2ui-button a2ui-button--${variant}`}
1010
data-a2ui-id={node.id}
1111
data-a2ui-component={node.component}
12+
disabled={ctx.readOnly}
1213
onClick={() => ctx.dispatchAction(node.action as A2uiAction | undefined, node)}
1314
>
1415
{ctx.render(node.child as string)}

frontend/src/a2ui/registry.ts

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ import type { A2uiAction, A2uiComponent, DynamicValue, SurfaceState } from "./ty
1111

1212
export interface RenderContext {
1313
surface: SurfaceState;
14+
readOnly: boolean;
1415
/** Render a child component by id. */
1516
render: (id: string | undefined) => ReactNode;
1617
/** Resolve a dynamic value against the surface data model. */

frontend/src/adk/client.ts

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2959,15 +2959,21 @@ export async function generateAgentDraftFromRequirement(
29592959

29602960
export async function createGeneratedAgentTestRun(
29612961
draft: AgentDraft,
2962-
runtime?: { runtimeId: string; region: string },
2962+
options: {
2963+
runtime?: { runtimeId: string; region: string };
2964+
modelCredentials?: Array<{ agentPath: number[]; apiKey: string }>;
2965+
comparisonId?: string;
2966+
} = {},
29632967
): Promise<GeneratedAgentTestRun> {
29642968
const res = await apiFetch("/web/generated-agent-test-runs", {
29652969
method: "POST",
29662970
headers: { "Content-Type": "application/json" },
29672971
body: JSON.stringify({
29682972
draft,
2969-
runtimeId: runtime?.runtimeId,
2970-
runtimeRegion: runtime?.region,
2973+
runtimeId: options.runtime?.runtimeId,
2974+
runtimeRegion: options.runtime?.region,
2975+
modelCredentials: options.modelCredentials ?? [],
2976+
comparisonId: options.comparisonId ?? "",
29712977
}),
29722978
});
29732979
if (!res.ok) {

0 commit comments

Comments
 (0)