Repository navigation
Conversation
Right after the first devin auth login, devin -p printed its welcome screen and exited 0 without running the prompt, and the job was recorded as completed with that text as the result. The new isOnboardingOutput helper (ANSI-stripped) detects it and the job is marked failed with a hint to run again. The check runs before normal rendering and is skipped when a sandbox hint applies, so the sandbox failure path is unchanged.
TheOnlyFusionCube
left a comment
There was a problem hiding this comment.
Reviewed against main and issue #4. The three-marker/ANSI detector is a good start and the integration test verifies failed job status. Please harden detection before merge: currently any successful task output that happens to quote all three onboarding phrases anywhere is classified as onboarding (false positive). Restrict matching to a recognizable welcome-screen shape, and add a negative test for a normal task/review that quotes the welcome text (and mixed output). Also gate onboarding classification on result.exitStatus === 0: currently a genuine failed run containing those phrases gets its failure replaced by an onboarding diagnosis. Preserve raw stdout/stderr and the original CLI exit status in the stored result; ensure the normal CLI exit-code contract is unchanged. The new rendered text hides the original welcome screen from result (though stored in payload.rawOutput); please include the original output in the user-visible result alongside the rerun hint. Add integration assertions for raw output retention and --json status, plus a negative case. HOL scanner logs identify the pre-existing one HIGH hardcoded-secret finding; #17 fixes it and is now merged. Refresh this branch, preserve #19's companion test block if it lands first, and rerun Test + HOL before merge.
Re-file of the former #8, as requested in #4, rebased onto current main.
Right after the first
devin auth login, seen on Devin CLI 3000.10.31 on macOS,devin -pprints its welcome screen and exits 0 without running the prompt. The job was then recorded as completed, with that text as Devin's answer.Changes
isOnboardingOutput(text)inlib/devin.mjs. It strips ANSI codes and requires all three of "Welcome to Devin CLI!", "Logged in as", and "Run ... devin ... to get started".runDevinJobrecords that output as a failed job with a "run the command again" hint, and keeps the raw output inpayload.rawOutput. The check is skipped when a sandbox hint applies, so the sandbox path from Fail closed when the Devin sandbox is unavailable #12 is unchanged.Tests: 6 unit tests in a new
tests/devin.test.mjs, plus 1 companion test. Full suite: 116/116.Note: this PR and the default-model re-file both append tests to the end of
tests/companion.test.mjs. Whichever merges second needs a trivial rebase that keeps both blocks; I'm happy to do it.