Skip to content

feat: SQLite task queue + worktree fanout + switch JSONC/verify-env - #92

Merged
Icaro0310 merged 4 commits into
mainfrom
feat/sqlite-task-queue
Oct 28, 2026
Merged

Icaro0310 merged 4 commits into
mainfrom
feat/sqlite-task-queue

Conversation

@Icaro0310

Copy link
Copy Markdown
Owner

Scout execution-brief items 12-14 and 16.

orchestrator queue (P2, 12-14): persistent SQLite queue reusing existing coordination contracts. States queued→claimed→running→succeeded|failed|cancelled; atomic claims via BEGIN IMMEDIATE (two-connection test proves exclusion); lease+heartbeat+orphan sweep (expired lease requeues until max_attempts then fails closed); idempotent --key resubmit; collect-before-report with incomplete stragglers + --strict; payload screened for secrets at write time; queue fanout with physical (repo,worktree)+(repo,branch) locks and file_collisions refusal; plan --files repeatable flag emitting plan units.

switch (P2, 16): JSONC trailing commas (string-aware strip, {,} and [1,,2] still fail loudly); verify-env with explicit PASS/WARN/FAIL per managed file, exit 1 on FAIL.

Tests: orchestrator 154 passed/27 subtests, switch 160 passed. ruff clean.

Icaro0310 and others added 2 commits October 27, 2026 16:49
Add queue_db.py: a local SQLite store (default
.devin-ecosystem/task-queue.db, same family as plans.jsonl) that turns
the orchestrator from planning-only into a durable coordination layer —
still no worker spawning or task execution, only recording and locking.

- States: queued -> claimed -> running -> succeeded|failed|cancelled
- Atomic claims: one BEGIN IMMEDIATE transaction does limit-check,
  select and update, so two workers can never claim the same task
- Leases: claims carry lease_expires_at renewed by heartbeat (first
  heartbeat also marks the task running); 'queue sweep' is the orphan
  recovery path — re-queues expired leases while attempts remain,
  fails them once max_attempts is exhausted
- Resource limits: --max-claims per worker (hard cap 3, mirroring
  MAX_WORKERS), bounded retries via --max-attempts, resubmission-safe
  --key idempotency
- collect-before-report: 'queue collect' gathers terminal results,
  stamps collected_at so a second call returns only new finishes, and
  reports 'incomplete' stragglers so no aggregate report can omit them
- No secrets: payloads/results/errors are screened for token-shaped
  material at write time and refused — the queue never stores
  credentials

'queue fanout <repo> --branch NAME' submits a unit targeting an
isolated git worktree: physical (repo,worktree)+(repo,branch) locks
inside the task-insert transaction, released only on terminal states
(re-queued retries keep their assignment), plus file_collisions-style
refusal against live tasks on the same repo. Does not run
git worktree add — the claiming worker creates it at the recorded path.

'plan --files' declares plan units directly as a file list (one
constrained unit per file echoed under units, independent_units set,
same collision check; standalone or alongside a spec; mutually
exclusive with units_detail and --from-triggers).

CLI: queue submit|fanout|claim|heartbeat|done|fail|cancel|sweep|collect|status
Pattern ported from a proven private reference implementation, adapted
to this package's conventions. 48 new tests; suite 154 passed.

Generated with [Devin](https://devin.ai) & QA Engineer Icaro Galvao

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Two consistency upgrades for the files devin-switch manages:

- jsonc.py now tolerates a comma that directly closes a ]/} container
  (the documented grammar now matches what VS Code and Devin itself
  accept). The stripper only removes a comma whose previous
  significant character is a value/closing token and whose next
  non-whitespace character is ] or }: commas inside strings are never
  touched, and {,} / [1,,2] / '{"a":,}' stay malformed and fail
  loudly. A second string-aware pass runs after comment stripping.

- 'devin-switch verify-env' is the strict gate alongside doctor:
  per managed file (config.json, mcp_config.json, hooks.v1.json,
  User/settings.json) one explicit PASS/WARN/FAIL line covering
  parseability, required keys (mcp_config.json must carry
  mcpServers), the documented hooks/MCP shapes, and referenced paths
  (mcpServers.*.command that looks like a filesystem path, and cwd,
  must exist; bare PATH binaries skip the check). Any FAIL exits 1.
  credentials.toml is still presence-checked only, never opened.
  Read-only, deterministic — same roots, same results.

32 new parser/verify tests (incl. a 9-case malformed-input matrix);
suite 160 passed.

Generated with [Devin](https://devin.ai) & QA Engineer Icaro Galvao

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@Icaro0310
Icaro0310 enabled auto-merge (squash) October 27, 2026 16:50
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Oct 27, 2026
…tch source literals

Runtime values are unchanged (still canonical fakes like AWS's documented AKIAIOSFODNN7EXAMPLE); only the source representation avoids the scanner's regexes.

Generated with [Devin](https://devin.ai) & QA Engineer Icaro Galvao

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@Icaro0310 Icaro0310 closed this Oct 27, 2026
auto-merge was automatically disabled October 27, 2026 19:14

Pull request was closed

@Icaro0310 Icaro0310 reopened this Oct 27, 2026
@Icaro0310 Icaro0310 closed this Oct 27, 2026
@Icaro0310 Icaro0310 reopened this Oct 27, 2026
@Icaro0310
Icaro0310 enabled auto-merge (squash) October 27, 2026 20:06
@Icaro0310 Icaro0310 closed this Oct 28, 2026
auto-merge was automatically disabled October 28, 2026 08:50

Pull request was closed

@Icaro0310 Icaro0310 reopened this Oct 28, 2026
@Icaro0310 Icaro0310 closed this Oct 28, 2026
@Icaro0310 Icaro0310 reopened this Oct 28, 2026
@github-actions

Copy link
Copy Markdown

Reviewer B (local model qwen2.5-coder:1.5b) — verdict: request_changes

The provided code snippet is a Python script for a CLI tool called devin-orchestrator. It includes various functionalities such as planning, queuing, and managing tasks. The script is designed to be used in a Devin workspace, which is a development environment for Python projects. The tool is used to manage the execution of tasks, including API, UI, and tests, and to coordinate the execution of these tasks across multiple workers.

Here's a breakdown of the key components and functionalities:

  1. Planning:

    • The plan command is used to compute a worker plan based on a JSON specification. The specification includes details such as the kind of task (implementation), the number of independent units, whether the task needs write access, the estimated scope, and a summary.
    • The --explain option provides a detailed explanation of the plan.
  2. Queueing:

    • The queue command is used to manage the task queue. It supports various operations such as submitting a unit, claiming a unit, leasing a unit, collecting units, sweeping orphaned units, collecting units before the aggregate report, and checking the status of a task.
    • The --files option is used to specify the files that the unit touches, which helps in avoiding collisions with live tasks.
  3. Installing Skills and Rules:

    • The script installs the devin-orchestrator skill and always-on rule into a Devin workspace. This allows the skill to be used in the Devin environment.
  4. Behavioral Matrix Testing:

    • The pytest -q command is used to run a behavioral matrix test suite. The suite includes tests for different scenarios, such as trivial, units, nested, and other combinations.
  5. Command-Line Interface:

    • The script provides a command-line interface with various subcommands such as plan, queue, install, and pytest. The --files

@Icaro0310
Icaro0310 merged commit f8d6c78 into main Oct 28, 2026
18 of 21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant