Skip to content

Skyvern demo: label simulated prompt + illustrative receipts (Tier 1) - #6

Open
aimable100 wants to merge 1 commit into
mainfrom
claude/angry-leavitt-380773
Open

Skyvern demo: label simulated prompt + illustrative receipts (Tier 1)#6
aimable100 wants to merge 1 commit into
mainfrom
claude/angry-leavitt-380773

Conversation

@aimable100

Copy link
Copy Markdown
Contributor

Summary

The Skyvern demo's attack mode scripts the agent (the prompt itself instructs it to pick ClearTone Ultra), and defended mode runs Tenuo authorization post-hoc against Skyvern's output rather than inline at each browser action. Both were plausible to a developer reading the code, but the blog/tutorial framed them as real inline interception. This PR closes the gap without changing the architecture — Tier 1 of the credibility-strategy doc. The defense story (compromised LLM, correct outcome, signed receipts) is unchanged; the framing now matches the code.

  • Runner — rename TASK_PROMPT_ATTACKTASK_PROMPT_ATTACK_SIMULATED. Add --live flag that runs attack/defended with the neutral comparison prompt so the DOM injection alone has to persuade the model. Tag the redirect/fallback/checkout receipts with synthetic=True; print_receipt_chain renders them as [ILLUSTRATIVE] with a header explaining the post-hoc reconstruction. Mode-aware "agent tricked" messaging.
  • Blog — new "How to read this demo" callout up front (simulated vs live, post-hoc disclosure), mode-aware Act 2 framing, [ILLUSTRATIVE] tags on receipts 7/8/9 with reasons, and a note under the architecture diagram about post-hoc vs Tier-2 inline. Fixes the browser_extract OneOf([...]) vs Wildcard() drift between blog and code.
  • Tutorial — documents --simulated vs --live; calls out post-hoc; labels illustrative receipts; "LLM tricked?" → "LLM compromised?" in the comparison table.
  • demo-outputsEDITED TRANSCRIPT — NOT A RAW RUN LOG header on all three files so the styled blog excerpts aren't mistaken for raw captures.

Tier 2 (inline integration at Skyvern's action handler) is intentionally not in this PR — that's a separate effort that requires forking/monkeypatching Skyvern.

Test plan

  • python skyvern-config/task.py clean — happy path unchanged
  • python skyvern-config/task.py attack — prints [SIMULATED] banner; selects ClearTone Ultra; ends with [SIMULATED] Agent selected ClearTone Ultra as scripted by the prompt
  • python skyvern-config/task.py attack --live — prints [LIVE] banner; neutral prompt; outcome depends on model
  • python skyvern-config/task.py defended --local — receipt chain shows [ILLUSTRATIVE] tags on receipts 7/8/9 and the post-hoc header
  • python skyvern-config/task.py --help reads cleanly and explains the modes
  • Blog and tutorial render correctly on GitHub (callouts, code blocks, table)

The attack mode's prompt scripts the agent to pick ClearTone Ultra, and
the defended mode runs Tenuo authorization post-hoc against Skyvern's
output rather than inline against each browser action. Both were
plausible to a developer reading the code but the blog/tutorial framed
them as real inline interception. This pass closes that gap without
changing the architecture:

- Rename TASK_PROMPT_ATTACK -> TASK_PROMPT_ATTACK_SIMULATED; add --live
  flag that runs attack/defended with the neutral comparison prompt so
  the DOM injection alone has to persuade the model.
- Tag the redirect/fallback/checkout receipts with synthetic=True;
  print_receipt_chain renders them as [ILLUSTRATIVE] with a header
  explaining the post-hoc reconstruction.
- Blog: new "How to read this demo" callout, mode-aware Act 2 framing,
  [ILLUSTRATIVE] tags on receipts 7/8/9, and a note under the
  architecture diagram about post-hoc vs Tier-2 inline integration.
  Fixes the browser_extract OneOf/Wildcard drift between blog and code.
- Tutorial: documents --simulated vs --live; calls out post-hoc; labels
  illustrative receipts; updates the three-runs comparison table.
- docs/demo-outputs: add an "EDITED TRANSCRIPT - NOT A RAW RUN LOG"
  header so the styled blog excerpts can't be mistaken for captures.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant