Skip to content

Treat the CLI's first-run welcome screen as a failed run - #20

Open
hessabi wants to merge 1 commit into
TheOnlyFusionCube:mainfrom
hessabi:fix/first-run-onboarding-v2
Open

hessabi wants to merge 1 commit into
TheOnlyFusionCube:mainfrom
hessabi:fix/first-run-onboarding-v2

Conversation

@hessabi

@hessabi hessabi commented Oct 25, 2026

Copy link
Copy Markdown
Contributor

Re-file of the former #8, as requested in #4, rebased onto current main.

Right after the first devin auth login, seen on Devin CLI 3000.10.31 on macOS, devin -p prints its welcome screen and exits 0 without running the prompt. The job was then recorded as completed, with that text as Devin's answer.

Changes

  • New exported isOnboardingOutput(text) in lib/devin.mjs. It strips ANSI codes and requires all three of "Welcome to Devin CLI!", "Logged in as", and "Run ... devin ... to get started".
  • runDevinJob records that output as a failed job with a "run the command again" hint, and keeps the raw output in payload.rawOutput. The check is skipped when a sandbox hint applies, so the sandbox path from Fail closed when the Devin sandbox is unavailable #12 is unchanged.
  • Exit-code policy is not touched. The tests assert job status and hint, not an exit code.

Tests: 6 unit tests in a new tests/devin.test.mjs, plus 1 companion test. Full suite: 116/116.

Note: this PR and the default-model re-file both append tests to the end of tests/companion.test.mjs. Whichever merges second needs a trivial rebase that keeps both blocks; I'm happy to do it.

Right after the first devin auth login, devin -p printed its welcome
screen and exited 0 without running the prompt, and the job was recorded
as completed with that text as the result. The new isOnboardingOutput
helper (ANSI-stripped) detects it and the job is marked failed with a
hint to run again. The check runs before normal rendering and is skipped
when a sandbox hint applies, so the sandbox failure path is unchanged.

@TheOnlyFusionCube TheOnlyFusionCube left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed against main and issue #4. The three-marker/ANSI detector is a good start and the integration test verifies failed job status. Please harden detection before merge: currently any successful task output that happens to quote all three onboarding phrases anywhere is classified as onboarding (false positive). Restrict matching to a recognizable welcome-screen shape, and add a negative test for a normal task/review that quotes the welcome text (and mixed output). Also gate onboarding classification on result.exitStatus === 0: currently a genuine failed run containing those phrases gets its failure replaced by an onboarding diagnosis. Preserve raw stdout/stderr and the original CLI exit status in the stored result; ensure the normal CLI exit-code contract is unchanged. The new rendered text hides the original welcome screen from result (though stored in payload.rawOutput); please include the original output in the user-visible result alongside the rerun hint. Add integration assertions for raw output retention and --json status, plus a negative case. HOL scanner logs identify the pre-existing one HIGH hardcoded-secret finding; #17 fixes it and is now merged. Refresh this branch, preserve #19's companion test block if it lands first, and rerun Test + HOL before merge.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants