Skyvern demo: label simulated prompt + illustrative receipts (Tier 1) - #6
Open
aimable100 wants to merge 1 commit into
Open
Skyvern demo: label simulated prompt + illustrative receipts (Tier 1)#6aimable100 wants to merge 1 commit into
aimable100 wants to merge 1 commit into
Conversation
The attack mode's prompt scripts the agent to pick ClearTone Ultra, and the defended mode runs Tenuo authorization post-hoc against Skyvern's output rather than inline against each browser action. Both were plausible to a developer reading the code but the blog/tutorial framed them as real inline interception. This pass closes that gap without changing the architecture: - Rename TASK_PROMPT_ATTACK -> TASK_PROMPT_ATTACK_SIMULATED; add --live flag that runs attack/defended with the neutral comparison prompt so the DOM injection alone has to persuade the model. - Tag the redirect/fallback/checkout receipts with synthetic=True; print_receipt_chain renders them as [ILLUSTRATIVE] with a header explaining the post-hoc reconstruction. - Blog: new "How to read this demo" callout, mode-aware Act 2 framing, [ILLUSTRATIVE] tags on receipts 7/8/9, and a note under the architecture diagram about post-hoc vs Tier-2 inline integration. Fixes the browser_extract OneOf/Wildcard drift between blog and code. - Tutorial: documents --simulated vs --live; calls out post-hoc; labels illustrative receipts; updates the three-runs comparison table. - docs/demo-outputs: add an "EDITED TRANSCRIPT - NOT A RAW RUN LOG" header so the styled blog excerpts can't be mistaken for captures.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The Skyvern demo's
attackmode scripts the agent (the prompt itself instructs it to pick ClearTone Ultra), anddefendedmode runs Tenuo authorization post-hoc against Skyvern's output rather than inline at each browser action. Both were plausible to a developer reading the code, but the blog/tutorial framed them as real inline interception. This PR closes the gap without changing the architecture — Tier 1 of the credibility-strategy doc. The defense story (compromised LLM, correct outcome, signed receipts) is unchanged; the framing now matches the code.TASK_PROMPT_ATTACK→TASK_PROMPT_ATTACK_SIMULATED. Add--liveflag that runs attack/defended with the neutral comparison prompt so the DOM injection alone has to persuade the model. Tag the redirect/fallback/checkout receipts withsynthetic=True;print_receipt_chainrenders them as[ILLUSTRATIVE]with a header explaining the post-hoc reconstruction. Mode-aware "agent tricked" messaging.[ILLUSTRATIVE]tags on receipts 7/8/9 with reasons, and a note under the architecture diagram about post-hoc vs Tier-2 inline. Fixes thebrowser_extractOneOf([...])vsWildcard()drift between blog and code.--simulatedvs--live; calls out post-hoc; labels illustrative receipts; "LLM tricked?" → "LLM compromised?" in the comparison table.EDITED TRANSCRIPT — NOT A RAW RUN LOGheader on all three files so the styled blog excerpts aren't mistaken for raw captures.Tier 2 (inline integration at Skyvern's action handler) is intentionally not in this PR — that's a separate effort that requires forking/monkeypatching Skyvern.
Test plan
python skyvern-config/task.py clean— happy path unchangedpython skyvern-config/task.py attack— prints[SIMULATED]banner; selects ClearTone Ultra; ends with[SIMULATED] Agent selected ClearTone Ultra as scripted by the promptpython skyvern-config/task.py attack --live— prints[LIVE]banner; neutral prompt; outcome depends on modelpython skyvern-config/task.py defended --local— receipt chain shows[ILLUSTRATIVE]tags on receipts 7/8/9 and the post-hoc headerpython skyvern-config/task.py --helpreads cleanly and explains the modes