Veta

Autonomous AI agent swarm for visual, functional, and accessibility testing of Android apps and mobile web — 100% of AI inference runs on AMD GPUs (Fireworks AI on AMD Instinct / self-hosted vLLM on ROCm), with containerized Android instances running locally.

Overview • Features • Architecture • How it works • API • Getting started • Project structure • Tech stack

Built for the AMD Developer Hackathon ACT II — Unicorn Track. No benchmarks, no constraints, just build: give an AI model eyes and hands, and let it QA your Android apps.

Overview

Veta replaces brittle, scripted UI tests with an AI-driven QA agent that watches an Android screen, decides what to do next based on a plain-English task description, and executes actions via ADB. Each run produces a pass/fail verdict, a complete action trace, and video/log artifacts — the same output a human QA tester would produce, at machine speed.

Sessions run in parallel across a fleet of containerized Android instances (redroid), managed by a scheduler with a warm pool and autoscaling. A web dashboard lets humans launch sessions, watch live screen mirrors, and review results.

Input type	Description
APK	Upload an Android APK, install it on a clean Android instance, and run a task against the native app
Web domain	Provide a URL to test inside a sandboxed mobile browser, with optional login credentials
Context file	Attach `.txt`, `.md`, `.docx`, or `.pdf` checklist alongside the task — contents are folded into the agent's working prompt at runtime

The system starts at ~10 concurrent sessions on a single host and scales to ~100 concurrent sessions on Kubernetes.

100% AMD-Powered AI

Every single model call — planning, execution, verification, reporting — runs on AMD GPUs. Zero inference happens on non-AMD hardware by default.

Path	What runs on AMD	Hardware
Fireworks AI (default)	Planner, executor, verifier, reporter	AMD Instinct MI250/MI300X
Self-hosted vLLM + ROCm	Same pipeline, your hardware	Any AMD GPU via ROCm

The agent has no fallback to non-AMD inference providers. Gemini, OpenRouter, and Anthropic are listed as optional alternatives in the settings UI — none are enabled or used by default. Out of the box, 100% of AI compute is AMD.

Features

Sub-agent architecture — Planner creates a checklist before execution. Executor drives per-step decisions. Verifier reviews the verdict before finalizing. Reporter writes a structured post-run analysis. Each is independently promptable and swappable.
Fixed action vocabulary — tap, long_press, swipe, scroll, drag, type, type_multi, key, back, dismiss_keyboard, wait, assert, done, request_manual_control, note. The model never emits raw shell commands.
Ground-truth screen tracking — Per-step screen_changed flag from hierarchy hash comparison, plus Android activity name monitoring via dumpsys. The model knows "which screen am I on" independently of visual appearance.
Toast interception — Post-action polling of the accessibility hierarchy catches system Toasts (~2-3.5s window) with code-level certainty, not visual guesswork.
Loop detection — Tracks repeated identical actions with no screen_changed effect; auto-inserts back after 3 repeats to break stuck states.
Verification gate — Before accepting a done verdict, a second model pass (verifier) checks every planned checkpoint against the action history. Can send the session back for more exploration if evidence is thin.
Post-run analysis — Reporter sub-agent generates a structured write-up: executive summary, checkpoint breakdown, technical explanation, severity rating, remediation recommendations. Available via API and PDF/CSV export.
Containerized Android fleet — Redroid containers with per-container binder device slots, health checks, warm pool management, and clean-state teardown between sessions.
Real-time streaming — WebSocket for log entries, step records, status changes, and live device screen mirror during a running session. Degrades to polling fallback if WebSocket fails.
Step replay player — Focus view and filmstrip view with touch marker animation, screenshot browsing, and auto-play.
Export — PDF report (jsPDF with professional layout, verdict, metadata, checkpoint breakdown, steps table, logs table) and CSV log export.
Session controls — End (stop + cancel) and restart running sessions from the UI.
Device identity profiles — Pixel 7 Pro, Pixel 6, Samsung S24, and more — spoofed via ro.build properties for realistic testing.
100% AMD-powered AI — Fireworks AI on AMD Instinct (default) or self-hosted vLLM on AMD ROCm. Every call_model_json() in the agent pipeline hits AMD GPUs.

Architecture

                      +------------------+
      People / CI --> |  New Session     |  APK / URL + task description
                      |  Input Form      |--------------------------------+
                      +------------------+                                |
                                                                          v
                      +------------------+                      +-------------------+
                      |  Fleet Dashboard |  <--------------->   |  Session          |
                      |  (live overview) |                      |  Scheduler        |
                      +------------------+                      +--------+----------+
                                                                          |
                      +-----------------------------------------------------+------------------------------------------+
                      v                          v                          v                                          v
               redroid container           redroid container           redroid container                    ... up to ~100
               + agent sidecar             + agent sidecar             + agent sidecar
               (observe->decide->act)      (observe->decide->act)      (observe->decide->act)
                      |                          |                          |
                      +---------> Object storage (screenshots, step JPEGs, logs)
                      +---------> Results DB (Postgres — sessions, steps, log entries, analyses, devices, settings, users)

Components

Component	Responsibility
New Session form	Accept APK upload or domain URL, task description, device/browser profile, optional context file
Fleet Dashboard	Live overview of every session with status bar, search, filter, sort, pagination
Session Detail	Live screen mirror + structured log stream + replay player + AI analysis panel
Session Scheduler	Dispatch loop (2s tick) + fleet loop (10s tick): queued sessions -> reserve/boot device -> dispatch agent
Android container (redroid)	One isolated Android 11 instance per session via Docker, with per-container binder device slots
Agent sidecar	Observe -> decide -> act loop. Sub-agents: planner, executor, verifier, reporter
Device Manager	Container lifecycle: provision, deploy, health-check, teardown, warm pool maintenance
Postgres DB	Source of truth for session metadata, steps, log entries, device state, session analyses, settings, users
File storage	Per-step replay screenshots (downscaled JPEG), uploaded APKs, stored on local filesystem

How it works

Agent loop

Each session runs a single sidecar process that cycles through five steps:

Observe — Capture a screenshot (adb screencap via adbutils) and the UI element hierarchy (uiautomator dump parsed into a flat element list with bounds, text, content-desc, and class).
Decide — Send image + hierarchy + task description + action history to the AI model. The model returns one structured action from the fixed vocabulary.
Act — Translate the action into ADB calls: input tap, input swipe, input text, input keyevent, etc.
Settle — Poll for system Toasts in the ~1s post-action window, then capture a fresh screenshot for the next step.
Ground truth check — Per-step hierarchy hash detects whether the screen actually changed; activity name tracking detects navigation independently of visual appearance.

The model must explicitly emit assert (expected vs. actual comparison) and done (pass/fail verdict with reasoning) — it cannot wander indefinitely.

Sub-agent pipeline

Task + screenshot
    |
    v
+------------------+
|  Planner         |  Creates 3-6 concrete checkpoints from task + screenshot (+ APK manifest activities)
|  (once)          |  Degrades gracefully to empty list -> executor/verifier fall back to free-form task
+--------+---------+
         | plan checklist
         v
+------------------+
|  Executor        |  Per-step: screenshot + hierarchy + history -> one action
|  (per step)      |  Retries on bad generation (3 attempts). Loop detection. Activity tracking.
+--------+---------+
         | "done" with proposed verdict
         v
+------------------+
|  Verifier        |  Checkpoint-by-checkpoint review against action history + final screenshot
|  (once)          |  Can set should_continue=true -> executor keeps going. Falls back to executor's verdict.
+--------+---------+
         | final verdict
         v
+------------------+
|  Reporter        |  Post-verdict write-up: summary, breakdown, technical explanation, severity, recommendations
|  (once)          |  Persisted to DB. Serves Analysis tab + PDF export. Degrades to checkpoint-derived fallback.
+------------------+

Session lifecycle

User submits an APK (or URL), task description, and optional context file through the New Session form.
API creates a queued session row with sequential ID (e.g. S9403).
Session Scheduler's dispatch loop (every 2s) picks up queued sessions up to the parallelism limit.
Scheduler reserves an idle device or cold-provisions one (binds a port + binder slot, boots redroid container, waits for adb + sys.boot_completed).
Session transitions booting -> initializing -> running. The agent sidecar installs the APK (if applicable), runs the planner, then begins the observe-decide-act loop.
When the executor emits done, the verifier reviews the evidence. If it passes, the session transitions to analyzing (device is released immediately), the reporter generates the write-up, then the session settles into passed or failed.
Artifacts (step screenshots, action log, analysis) are persisted. The container is torn down. The fleet loop (every 10s) re-deploys the device for the warm pool.
The Fleet Dashboard and Session Detail reflect the final verdict and analysis in real time.

Session statuses

Status	Meaning
`queued`	Waiting for an available device slot
`booting`	Android container is starting up
`initializing`	Installing APK / preparing agent
`running`	Agent loop is actively executing the task
`awaiting_questions`	Review mode: waiting for user answers to clarifying questions
`awaiting_approval`	Review mode: waiting for user to approve the generated test plan
`analyzing`	Verdict reached; reporter generating write-up
`passed`	Task completed successfully
`failed`	Task completed but assertions failed
`stopped`	Manually stopped by a user
`error`	Infrastructure or system error
`timed_out`	Agent exceeded max steps (50)

Device model

Each redroid container runs Android 11 (1080x2400, 480dpi) with nav bar + soft keyboard disabled via policy_control and pm disable-user.
Devices are identified by D-{hex} ID, assigned a host port (5555-5655 range) and a binder slot (binder/hwbinder/vndbinder triplet) for the kernel's binder driver — a hard cap set at modprobe time on the host.
Warm pool: idle, healthy devices maintained up to a user-configurable target. Devices in error state are reaped after 30s backoff.
Health checks run every 10s: container running + sys.boot_completed.
Device identity profiles (Pixel 7 Pro / Pixel 6 / Samsung S24) are applied via ro.build property overrides before any app touches the device.

API

The backend is a Python FastAPI server at /api. Interactive docs at http://localhost:8000/docs.

Sessions

Method	Path	Purpose
`POST`	`/api/sessions`	Create a new session (multipart form: APK file or URL + task + device profile)
`GET`	`/api/sessions`	List sessions (cursor-paginated, filterable by status/device/build/text search)
`GET`	`/api/sessions/stats`	Dashboard stats (running/queued/passed/failed counts + success rate)
`GET`	`/api/sessions/{id}`	Get single session details
`PATCH`	`/api/sessions/{id}/control`	Control session (action: `end` or `restart`)
`POST`	`/api/sessions/{id}/answers`	Submit answers to review clarifying questions
`POST`	`/api/sessions/{id}/approve`	Approve review-mode test plan and queue session
`POST`	`/api/sessions/{id}/regenerate-plan`	Regenerate test plan from existing answers
`POST`	`/api/sessions/{id}/back-to-questions`	Go back to clarifying questions from plan review
`GET`	`/api/sessions/{id}/logs`	Get log entries with cursor (`since`)
`POST`	`/api/sessions/{id}/logs`	Create a log entry
`GET`	`/api/sessions/{id}/steps`	Get replay steps with cursor (`since_step`)
`GET`	`/api/sessions/{id}/steps/{n}/screenshot`	Get step screenshot (JPEG)
`GET`	`/api/sessions/{id}/analysis`	Get session analysis write-up
`POST`	`/api/sessions/{id}/analysis/regenerate`	Regenerate analysis for a finished session
`WS`	`/api/sessions/{id}/events`	WebSocket — real-time log/step/status deltas
`WS`	`/api/sessions/{id}/stream`	WebSocket — live screen frame stream (PNG)

Devices

Method	Path	Purpose
`GET`	`/api/devices`	List all devices
`POST`	`/api/devices`	Provision a new device (boots redroid container)
`GET`	`/api/devices/capacity`	Binder slot usage (total/used/available)
`POST`	`/api/devices/{id}/start`	Start an offline device
`POST`	`/api/devices/{id}/stop`	Stop a running device
`POST`	`/api/devices/{id}/restart`	Restart a device (re-deploy container)
`POST`	`/api/devices/{id}/maintenance`	Toggle maintenance mode
`POST`	`/api/devices/{id}/warm`	Adjust warm instance count (delta -8 to +8)
`DELETE`	`/api/devices/{id}`	Remove device (tear down container, free port/slot)

Auth

Method	Path	Purpose
`POST`	`/api/auth/login`	Login (email + password)
`POST`	`/api/auth/signup`	Register new user
`POST`	`/api/auth/logout`	Logout

Settings

Method	Path	Purpose
`GET`	`/api/settings`	Get app settings (workspace, region, notifications, parallelism, API key, warm pool size, max devices)
`PUT`	`/api/settings`	Update app settings

Getting started

Prerequisites

Docker Engine and Docker Compose (for Postgres and redroid containers)
Python 3.11+
Node.js 20+
An AI provider API key (Fireworks AI on AMD Instinct recommended, or any OpenAI-compatible provider)

One-command start

Development (hot reload):

./dev.sh

Production (built frontend, no reload):

./start.sh

Both scripts start Postgres (if not running), run migrations, and launch the full stack. Ctrl-C stops everything.

Step by step

1. Configure environment

cd backend
cp .env.example .env

Edit .env to set your AI_API_KEY and any other settings.

2. Start Postgres

docker compose up -d

Or manually:

docker run -d --name veta-pg \
  -e POSTGRES_USER=postgres \
  -e POSTGRES_PASSWORD=postgres \
  -e POSTGRES_DB=veta \
  -p 5432:5432 \
  postgres:16-alpine

3. Backend

cd backend
pip install -e .
alembic upgrade head
uvicorn app.main:app --reload --host 0.0.0.0 --port 8000

The API starts at http://localhost:8000. Docs at http://localhost:8000/docs.

On first boot, the server auto-seeds admin user (admin/admin) and default settings.

4. Frontend

cd frontend
npm install
npm run dev

The dashboard starts at http://localhost:8080. Login with any email/password.

Set VITE_API_URL to change the backend address (default http://localhost:8000).

Redroid device fleet (optional — needed for real sessions)

# Load binder kernel module (adjust device count for your host)
sudo modprobe binder_linux devices="binder,hwbinder,vndbinder"

# Example for 3 concurrent devices:
# devices="binder,hwbinder,vndbinder,binder1,hwbinder1,vndbinder1,binder2,hwbinder2,vndbinder2"

# Ensure adb is installed and accessible
adb devices

# Set required env vars
export REDROID_IMAGE=redroid/redroid:11.0.0-latest

Environment variables

Variable	Default	Description
`DATABASE_URL`	`postgresql+asyncpg://postgres:postgres@localhost:5432/veta`	Postgres connection string
`REDROID_IMAGE`	`redroid/redroid:11.0.0-latest`	Android container image
`ADB_HOST`	`localhost`	ADB host address
`REDROID_PORT_START`	`5555`	Start of port range for device ADB
`REDROID_PORT_END`	`5655`	End of port range (up to ~100 devices)
`BINDER_SLOTS`	auto-detected from `/dev`	Override for binder slot count
`AI_API_KEY`	--	AI provider API key (also configurable via Settings UI)
`AI_PROVIDER`	`fireworks`	AI provider: `fireworks`, `veta-ai`, `google-gemini`, `openrouter`, `deepseek`, `anthropic`
`AI_MODEL`	provider default	Model name override
`MODEL_ENDPOINT`	provider default	API base URL override

Project structure

veta/
  Docs/                        Design documents, wireframes, logo
  server/                      vLLM inference server (AMD ROCm)
    veta_inference_server.ipynb  Notebook: serve vision models on AMD GPUs via vLLM
    bin/                         cloudflared binary for tunnel

  backend/                     FastAPI backend API
    app/
      main.py                  FastAPI entrypoint + lifespan (auto-seed, stale cleanup, dispatch + fleet loops)
      config.py                pydantic-settings (.env / env vars)
      database.py              Async SQLAlchemy engine + session factory
      dispatcher.py            Session dispatch loop + fleet health loop
      device_manager.py        Redroid container lifecycle, port/binder allocation, warm pool, health checks
      device_events.py         SSE device state change notifications
      apk_utils.py             APK metadata extraction (aapt + apkutils)
      screenshot_store.py      Per-step screenshot downscale + JPEG storage
      models/                  SQLAlchemy ORM models (session, step, log_entry, device, settings, session_analysis, user)
      schemas/                 Pydantic request/response schemas (camelCase JSON)
      routers/                 REST + WebSocket endpoint handlers (sessions, steps, logs, devices, auth, settings, stream)
      services/
        session_analysis.py    Build + store session analysis (bridges reporter sub-agent to DB)
    adapters/                  Port implementations (ApkInspectorImpl, etc.)
    alembic/                   Database migrations
    scripts/
      seed.py                  Seed admin user + default settings
    uploads/                   Uploaded APKs + step screenshot storage

  agent/                       AI agent sidecar
    agents/
      base.py                  Shared retry logic (generate_with_retries)
      planner.py               Pre-execution checkpoint builder
      executor.py              Per-step action decision
      verifier.py              Post-decision verification
      reporter.py              Post-verdict write-up
    core/
      adb.py                   adbutils device handle
      device.py                Screenshots, action execution, marker extraction, activity tracking
      hierarchy.py             XML hierarchy parsing, element signature hashing
      model_client.py          Multi-provider AI client (Gemini SDK + OpenAI-compat) with retry + JSON extraction
      apk.py                   APK metadata extraction, install/launch on device
      cancellation.py          Thread-safe session cancellation
      thinking_hooks.py        Real-time thinking indicator callbacks
      modes.py                 Mode configs (review/run) + allowed action groups
      skill_text.py            Loads agent playbook into system prompt
    ports/                     Protocol interfaces for dependency injection
      apk_inspector.py         ApkInspector protocol (get_all_activities, get_apk_details)
    skills/                    Agent playbook (loaded into system context)
      SKILL.md                 Master playbook — decision loop, confidence/targeting rules, verdict discipline
      actions.md               Full action vocabulary with semantics for each action type
      elements.md              UI element taxonomy — how to recognize and interact with different UI components
      scenarios.md             Step-by-step playbooks for authentication, permissions, onboarding, forms, search
      debugging.md             Recovery strategies for stuck states, loop detection, unexpected screens
    memory.py                  SessionMemory — action history, loop detection, activity tracking, screen stack
    orchestrator.py            Session state machine wiring all sub-agents together (booting -> ... -> passed/failed)
    phases.py                  Phase-based lifecycle (SetupPhase, ReviewPhase, RunPhase)
    runner.py                  Compatibility entrypoint (delegates to orchestrator)

  frontend/                    TanStack Start web dashboard
    src/
      routes/                  File-based routing
        __root.tsx             Root layout + sidebar + auth
        index.tsx              Fleet Dashboard (stat bar, search, filter, sort, pagination, session cards)
        session.$id.tsx        Session Detail (live mirror, log stream, replay player, analysis panel, export, filmstrip)
        new-session.tsx        New Session form (APK/URL toggle, file drag-drop, context attachment, device profile)
        devices.tsx            Device Farm (card grid, provision dialog, warm pool controls, binder tracking)
        history.tsx            Run History (table view, sort, filter)
        settings.tsx           Settings (workspace, execution, API key, notifications, integrations)
        login.tsx / signup.tsx Auth pages
      components/
        session-card.tsx       Fleet dashboard session card
        session-analysis-panel.tsx  Analysis tab UI
        ui/                    shadcn/ui primitives (50+ components)
      lib/
        api.ts                 API client (fetch + WebSocket wrappers, all endpoints)
        auth.ts                Session storage auth
        session-report.ts      PDF + CSV export (jsPDF with professional layout)
    public/                    Static assets (favicon, logo)

Agent architecture

Sub-agents

Located in agent/agents/. Each is independently promptable and can be retried or swapped without affecting the others.

Agent	File	Purpose
planner	`planner.py`	Builds 3-6 concrete checkpoints from task + first screenshot (+ APK manifest). Runs once before execution.
executor	`executor.py`	Per-step decision: screenshot + hierarchy + history -> one action. Retries on bad generation (3 attempts).
verifier	`verifier.py`	Checkpoint-by-checkpoint review against action history. Can send session back for more exploration.
reporter	`reporter.py`	Post-verdict structured write-up. Falls back to checkpoint-derived summary on model failure.

Core modules

Located in agent/core/.

Module	Purpose
`device`	Screenshots, action execution dispatch, activity tracking, marker extraction
`hierarchy`	UiAutomator hierarchy dump + parsing + element signature hashing
`model_client`	Multi-provider HTTP client (Gemini SDK + OpenAI-compat, Fireworks on AMD Instinct), JSON extraction, retry logic
`apk`	APK install, package discovery, app launch on device
`adb`	Shared adbutils device handle
`cancellation`	Thread-safe session cancellation registry
`modes`	Mode configs (review/run) + action group definitions
`skill_text`	Agent playbook loaded into the executor's system prompt

AI providers

Provider	SDK	Backend hardware	Models	Status
Fireworks AI	`openai`	AMD Instinct MI250/MI300X	`qwen3.7-plus`, `firefunction-v2`, Gemma 4	✅ Default (100% AMD)
Veta AI (self-hosted)	`openai`	AMD ROCm (vLLM)	Any HF vision model (Qwen2.5-VL, Gemma, Llama)	✅ 100% AMD
Google Gemini	`google-genai`	Google TPU	`gemma-4-26b-a4b-it`	⬜ Optional
OpenRouter	`openai`	Varies	`openai/gpt-4o-mini`, any routed model	⬜ Optional
Deepseek	`openai`	Varies	`deepseek-v4-flash`	⬜ Optional
Anthropic	`openai`	Varies	Anthropic models via OpenAI-compat endpoint	⬜ Optional

100% of inference runs on AMD out of the box. The two AMD-backed providers (Fireworks + self-hosted ROCm) handle every agent call. Gemini, OpenRouter, Deepseek, and Anthropic are optional alternatives — none are enabled by default.

Agent skills

The agent playbook (agent/skills/) is loaded into the executor's system context:

File	Purpose
`SKILL.md`	Master playbook — decision loop, confidence/targeting rules, verdict discipline
`actions.md`	Full action vocabulary with semantics for each action type
`elements.md`	UI element taxonomy — recognizing and interacting with buttons, sliders, toggles, dropdowns, dialogs
`scenarios.md`	Step-by-step playbooks for authentication, permissions, onboarding, forms, search
`debugging.md`	Recovery strategies for stuck states, loop detection, unexpected screens

Tech stack

Layer	Technology
Backend	Python 3.11+, FastAPI, SQLAlchemy (async), Alembic, PostgreSQL
Agent loop	Python, httpx, Pillow, google-genai SDK, openai SDK
Frontend	React 19, TanStack Start, TanStack Router, Tailwind CSS v4, shadcn/ui
AI inference (cloud, default)	Fireworks AI — AMD Instinct MI250/MI300X (100% AMD)
AI inference (self-hosted)	vLLM + AMD ROCm (100% AMD)
Android runtime	Docker, redroid (Android 11), ADB, adbutils, uiautomator
Infrastructure	Docker, Docker Compose (dev), Kubernetes (target)
Real-time	WebSocket (live frame stream + events bus), SSE fallback
Export	jsPDF + jspdf-autotable (PDF), CSV

Cost model

Approach	Cost per session (~50 steps)	When to use
Fireworks API (AMD Instinct)	~$0.02-0.05	Development, low-volume QA
Self-hosted vLLM on ROCm	GPU electricity only	Production, high-volume CI/CD
Gemini CachedContent	~$0.01 (95% cached)	Optional fallback / multi-provider

Name		Name	Last commit message	Last commit date
.github/ISSUE_TEMPLATE		.github/ISSUE_TEMPLATE
Docs		Docs
agent		agent
backend		backend
frontend		frontend
.gitignore		.gitignore
CHANGELOG.md		CHANGELOG.md
LICENSE		LICENSE
README.md		README.md
SECURITY.md		SECURITY.md
dev.sh		dev.sh
docker-compose.yml		docker-compose.yml
start.sh		start.sh

Latest commit

History

Folders and files

Repository files navigation