Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 31 additions & 17 deletions docs/codebase-walkthrough.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ A complete guide to understanding how FinalRun works, from CLI invocation to tes

## 1. Architecture Overview

FinalRun is a **monorepo with 5 packages** that together form an AI-powered mobile app testing tool:
FinalRun is a **monorepo with 7 packages** that together form an AI-powered mobile app testing tool:

```text
finalrun-ts/
Expand All @@ -33,7 +33,9 @@ finalrun-ts/
│ ├── cli/ CLI commands, test orchestration, report server, artifact writing
│ ├── goal-executor/ AI agent loop: screenshot → LLM plan → device action → repeat
│ ├── device-node/ Device management: gRPC driver, screenshots, action execution
│ └── report-web/ Next.js web app for viewing reports (alternative to built-in server)
│ ├── cloud-core/ Cloud-submission logic, bundled into the compiled CLI binary
│ ├── local-runtime/ Per-platform runtime asset bundle (driver APKs/iOS zips, proto, report SPA dist)
│ └── report-web/ React SPA (Vite) for viewing reports, served by the CLI's report server
```

**High-level flow:**
Expand All @@ -47,15 +49,17 @@ Report displayed Artifacts saved to disk AI executes test on device
(HTML in browser) <────────────────────────── (screenshot → LLM → tap/type → repeat)
```

### Why 5 packages?
### Why 7 packages?

| Package | Reason for separation |
|---------|----------------------|
| `common` | Types shared by all packages. Changing a type here forces all consumers to stay in sync. |
| `cli` | User-facing. Handles I/O, config, orchestration. Should not know about gRPC or LLM internals. |
| `goal-executor` | The AI loop is complex enough to deserve isolation. It could theoretically be swapped for a different execution strategy. |
| `device-node` | Platform-specific (Android/iOS) device code. Isolates gRPC, ADB, and driver concerns from business logic. |
| `report-web` | Optional Next.js frontend. The CLI has a built-in server too, so this is an enhancement, not a dependency. |
| `cloud-core` | Pure cloud-submission logic kept free of CLI I/O, so it can be bundled into the compiled CLI binary. |
| `local-runtime` | Builds the per-platform runtime asset tarball (driver APKs/iOS zips, gRPC proto, report SPA dist) the compiled CLI needs at runtime. |
| `report-web` | React SPA (Vite) frontend that the CLI's report server serves. Its Vite bundle is copied into `cli/dist/report-app` at build time (`copyReportApp.mjs`) and into the runtime tarball (`buildRuntimeTarball.mjs`) — both builds fail without it, so it is a build-time dependency of the CLI, not an optional alternative. |

---

Expand Down Expand Up @@ -188,7 +192,7 @@ finalrun doctor Check host readiness
finalrun test auth/login.yaml --env staging --platform android --model openai/gpt-5.4-mini
```

**What happens in `runTestCommand()` (bin/finalrun.ts:271-350):**
**What happens in `runTestCommand()` (bin/finalrun.ts:372-485):**

```text
1. normalizeTestSelectors() Split comma-separated selectors, trim whitespace
Expand All @@ -214,7 +218,7 @@ finalrun test auth/login.yaml --env staging --platform android --model openai/gp

## 4. Phase 1: Validation (`runCheck`)

**File:** `packages/cli/src/checkRunner.ts`
**File:** `packages/common/src/checkRunner.ts`

Before touching any device, FinalRun validates everything:

Expand Down Expand Up @@ -675,10 +679,10 @@ Then `rebuildRunIndex()` scans all run directories and regenerates `runs.json`

FinalRun has two ways to display reports:

| | Built-in Server (`cli`) | Next.js App (`report-web`) |
| | Built-in Server (`cli`) | React SPA (`report-web`) |
|---|---|---|
| **File** | `packages/cli/src/reportServer.ts` | `packages/report-web/` |
| **Technology** | Raw `http.createServer()` | Next.js App Router |
| **Technology** | Raw `http.createServer()` | Vite + React SPA |
Comment thread
coderabbitai[bot] marked this conversation as resolved.
| **When used** | Default (`finalrun start-server`) | Development or custom deploy |
| **Routes** | Same | Same |

Expand Down Expand Up @@ -1093,31 +1097,41 @@ Quick reference: where to find what.
| `TestDefinition.ts` | `TestDefinition`, `BindingReference` | What a test looks like after YAML parsing |
| `TestResult.ts` | `TestResult`, `AgentAction`, `FirstFailure`, `TestStatus` | What a test result looks like after execution |
| `SuiteDefinition.ts` | `SuiteDefinition` | What a suite looks like after YAML parsing |
| `DeviceAction.ts` | `TapAction`, `EnterTextAction`, `ScrollAbsAction`, ... (18 types) | All possible device actions |
| `DeviceAction.ts` | `TapAction`, `EnterTextAction`, `ScrollAbsAction`, ... (22 concrete action classes) | All possible device actions |
| `Environment.ts` | `AppConfig`, `EnvironmentConfig`, `RuntimeBindings` | Environment and binding types |
| `Trace.ts` | `AgentActionTrace`, `TraceSpan`, `TimingInfo` | Performance tracing types |
| `Hierarchy.ts` | `Hierarchy`, `HierarchyNode` | UI element tree from device |

### `packages/cli/src/`
### `packages/common/src/` (non-model modules)

| File | What it does | Key functions |
|------|-------------|---------------|
| `bin/finalrun.ts` | CLI entry point, command definitions | `runTestCommand()` |
| `testRunner.ts` | Main test orchestrator | `runTests()` |
| `checkRunner.ts` | Validation phase | `runCheck()` |
| `sessionRunner.ts` | Device setup + test execution | `prepareTestSession()`, `executeTestOnSession()` |
| `testLoader.ts` | YAML file parsing | `loadTest()`, `loadTestSuite()`, `loadEnvironmentConfig()` |
| `testSelection.ts` | Test file discovery | `selectTestFiles()`, `expandSelector()` |
| `testCompiler.ts` | Test → AI prompt | `compileTestObjective()` |
| `workspace.ts` | Workspace discovery | `resolveWorkspace()`, `loadWorkspaceConfig()` |
| `appConfig.ts` | App configuration | `resolveAppConfig()` |
| `env.ts` | Environment variables | `CliEnv`, `parseModel()`, `resolveApiKey()` |
| `env.ts` | Environment variables | `CliEnv` |
| `constants.ts` | Shared constants, model string parsing | `parseModel()`, `parseReasoningLevel()` |

### `packages/cli/bin/`

| File | What it does | Key functions |
|------|-------------|---------------|
| `finalrun.ts` | CLI entry point, command definitions | `runTestCommand()` |

### `packages/cli/src/`

| File | What it does | Key functions |
|------|-------------|---------------|
| `testRunner.ts` | Main test orchestrator | `runTests()` |
| `sessionRunner.ts` | Device setup + test execution | `prepareTestSession()`, `executeTestOnSession()` |
| `testCompiler.ts` | Test → AI prompt | `compileTestObjective()` |
| `apiKey.ts` | API key resolution | `resolveApiKey()` |
| `reportWriter.ts` | Artifact writing | `ReportWriter` class |
| `runIndex.ts` | Run index management | `rebuildRunIndex()`, `loadRunIndex()` |
| `reportServer.ts` | Built-in HTTP server | `serveReportWorkspace()` |
| `reportServerManager.ts` | Server lifecycle | `startOrReuseWorkspaceReportServer()` |
| `reportTemplate.ts` | Run detail HTML | `renderHtmlReport()` |
| `reportIndexTemplate.ts` | Run index HTML | `renderRunIndexHtml()` |
| `hostPreflight.ts` | SDK checks | `runHostPreflight()` |

### `packages/goal-executor/src/`
Expand Down
11 changes: 7 additions & 4 deletions docs/memory/common/env.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,16 +111,19 @@ the env module that no consumer actually needs.
carries a comment naming it a backward-compatibility shim. The block retains all nine symbols even
though no production module imports through it.
**Why**: `env.test.ts:6` imports `parseModel` and `parseReasoningLevel` from `../env.js`, and the
constitution's Test Integrity rule forbids editing a test to chase a moved import path — a test
conforms to the spec, and rewriting it to make a refactor compile inverts that. Keeping the block
re-export is what keeps that import path resolving — the shim exists for consumer compatibility with
the one line in the repo that still reaches these symbols through `env.js`. Keeping the block
also makes `env.js`'s export surface independent of where the validators are defined, which is what
lets a move like this one claim zero observable behavior change. The block's justification is exactly
that one import line: no production module imports anything but `CliEnv` from `env.js`, and
`MODEL_FORMAT_EXAMPLE`, `PROVIDER_ENV_VARS`, `SUPPORTED_AI_PROVIDERS`,
`SUPPORTED_AI_PROVIDERS_LABEL`, `ParsedModel`, and `SupportedProvider` have no `env.js` consumer
anywhere in the repo.
**Rejected**: (a) deleting the barrel and repointing importers — forces an import-path edit in
`env.test.ts`, which Test Integrity prohibits; (b) leaving a duplicate definition in `env.ts` so
**Rejected**: (a) deleting the barrel and repointing `env.test.ts`'s import at `../constants.js` —
an edit the constitution's Test Integrity rule permits (it allows updating a test to match the spec,
and prohibits only reshaping *implementation* to suit test infrastructure), but one that narrows
`env.js`'s export surface for no consumer's benefit, and that surface stability is what makes the
move's zero-observable-change claim checkable; (b) leaving a duplicate definition in `env.ts` so
the package root's export set stays literally identical — two definitions of the same validator is
exactly the drift a single home exists to prevent.
*Introduced by*: 260731-65sg-env-structural-refactor-pilot
Expand Down
15 changes: 11 additions & 4 deletions docs/memory/common/hierarchy.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
type: memory
description: "UI-hierarchy parse contract (`common/src/models/Hierarchy.ts`, consumed by 16 files in goal-executor/device-node): `fromJsonString` dispatches array→flat / object→tree, and the paths are deliberately not equivalent — only the flat path shortens `:id/` ids, infers `isImage` from the class, takes `identifier`; alias resolution is `??`-presence via `_pick`/`orDefault`, never truthiness, so `false` and `''` survive; bounds are a 4-array or a left/top/right/bottom object; reads are unvalidated casts."
description: "UI-hierarchy parse contract (`common/src/models/Hierarchy.ts`, consumed by four goal-executor files): `fromJsonString` dispatches array→flat / object→tree, and the paths are deliberately not equivalent — only the flat path shortens `:id/` ids, infers `isImage` from the class, takes `identifier`; alias resolution is `??`-presence via `_pick`/`orDefault`, never truthiness, so `false` and `''` survive; bounds are a 4-array or a left/top/right/bottom object; reads are unvalidated casts."
---
# UI Hierarchy Parsing (common)

Expand All @@ -10,9 +10,16 @@ description: "UI-hierarchy parse contract (`common/src/models/Hierarchy.ts`, con

`packages/common/src/models/Hierarchy.ts` turns the driver's UI-hierarchy JSON into `HierarchyNode`s
and the flattened list that `toPromptElementsForPlanner` / `toPromptElementsForGrounder` build
planner and grounder prompts from. It is the most-depended-on parser in the repo — 16 files across
`goal-executor` and `device-node` consume `Hierarchy`/`HierarchyNode` — and its one production entry
point is `Hierarchy.fromJsonString`, called on the device-capture path in
planner and grounder prompts from. Every consumer of `Hierarchy`/`HierarchyNode` lives in
`goal-executor` — `ai/AIAgent.ts`, `ActionExecutor.ts`, `TestExecutor.ts` and
`GrounderResponseConverter.ts`, each importing through the `@finalrun/common` barrel.
**`device-node` consumes neither type**, even though it is the client that fetches every payload
(`GrpcDriverClient.getHierarchy` / `getScreenshotAndHierarchy`) and owns a
`DeviceScreenshotAndHierarchy` interface of its own: the hierarchy crosses that package as an opaque
JSON `string` — the declared field type in both `GrpcDriverClient`'s response and
`DeviceRuntime.DeviceScreenshotAndHierarchy` — and only `goal-executor` parses it. So a grep for
`Hierarchy` under `device-node` hits RPC names and that local interface, never this type.
The one production entry point is `Hierarchy.fromJsonString`, called on the device-capture path in
`packages/goal-executor/src/TestExecutor.ts`. A behaviour change here surfaces as a grounding
failure at runtime rather than as a test failure, so the parse contract is pinned by
mutation-verified characterization tests in `packages/common/src/models/test/Hierarchy.test.ts`
Expand Down
2 changes: 1 addition & 1 deletion docs/memory/common/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,4 +8,4 @@ description: "`packages/common` — the base of the dependency graph, imported b
| File | Description |
|------|-------------|
| [env](env.md) | `CliEnv` environment loading (`packages/common/src/env.ts`): `load` layers `.env.<envName>` → plain `.env` (fill-only) → OS env (highest precedence), `includeDotEnv` opts out only on a literal `false`, and `getRequired`'s falsy check makes an empty string as missing as an absent key. Model and reasoning-level validation live in `constants.ts` beside their level lists, while `env.ts`'s re-export block is a deliberate backward-compat shim for the one import path that still routes through it. |
| [hierarchy](hierarchy.md) | UI-hierarchy parse contract (`common/src/models/Hierarchy.ts`, consumed by 16 files in goal-executor/device-node): `fromJsonString` dispatches array→flat / object→tree, and the paths are deliberately not equivalent — only the flat path shortens `:id/` ids, infers `isImage` from the class, takes `identifier`; alias resolution is `??`-presence via `_pick`/`orDefault`, never truthiness, so `false` and `''` survive; bounds are a 4-array or a left/top/right/bottom object; reads are unvalidated casts. |
| [hierarchy](hierarchy.md) | UI-hierarchy parse contract (`common/src/models/Hierarchy.ts`, consumed by four goal-executor files): `fromJsonString` dispatches array→flat / object→tree, and the paths are deliberately not equivalent — only the flat path shortens `:id/` ids, infers `isImage` from the class, takes `identifier`; alias resolution is `??`-presence via `_pick`/`orDefault`, never truthiness, so `false` and `''` survive; bounds are a 4-array or a left/top/right/bottom object; reads are unvalidated casts. |
4 changes: 2 additions & 2 deletions docs/memory/device-node/log-capture.md
Original file line number Diff line number Diff line change
Expand Up @@ -164,8 +164,8 @@ and `finalizeQuietly`'s own `Logger.e` is guarded for the same reason: `finalize
recorded error is what makes that `catch` reachable on the very failure the guard exists for, and
`finalizeQuietly` MUST resolve for callers that are already returning a failure. The reason is
`Logger.e`'s **independent** fallibility: `Logger._emit`'s sink
loop (`packages/common/src/logger.ts:103-105`) runs every sink with no `try`/`catch`, and the CLI
installs `ReportWriter.createLoggerSink()` (`packages/cli/src/reportWriter.ts:132`), a bare
loop (`packages/common/src/logger.ts`) runs every sink with no `try`/`catch`, and the CLI
installs `ReportWriter.createLoggerSink()` (`packages/cli/src/reportWriter.ts`), a bare
synchronous `fs.appendFileSync`, so a full disk, a permissions change or a removed artifacts
directory makes the log call throw on its own schedule. The two failures can also be one, but only
conditionally: the device log lives at `<os.tmpdir()>/finalrun-logs/…` and the runner log at
Expand Down
17 changes: 13 additions & 4 deletions docs/memory/drivers/grpc-contract.md
Original file line number Diff line number Diff line change
Expand Up @@ -124,10 +124,19 @@ per tick, for a request that asked for 24 frames a second. Both drivers state th
computation lives: Android's `TestUtils.calculateFrameDelay` uses `1000.0 / fps.toDouble()`, and the
Swift `XCViewHierarchyManager` cites it.

`fps` MUST be clamped to `1...60` (`streamingFpsRange`), matching Android's `coerceIn(1, 60)`: below 1
the interval is unbounded, above 60 the hierarchy snapshot cannot keep up. An omitted `fps` defaults
to `1` (`defaultStreamingFps`) — the proto's documented 24 is not adopted, because changing the
default is a separate behaviour change from fixing the arithmetic.
`fps` MUST be clamped to `1...60` (`streamingFpsRange`), matching Android's `coerceIn(1, 60)`. The
clamp guards the divisor: in `GrpcDriverServer.swift`'s integer `1_000_000_000 / fps`, an fps of `0`
is a division by zero and a negative fps a negative-to-`UInt64` conversion — both Swift traps that
kill the XCUITest runner — while the floating-point paths (`calculateFrameDelay`,
`XCViewHierarchyManager`) turn an fps below 1 into an unbounded or negative interval. The upper
bound of 60 is the chosen cap on hierarchy-snapshot load shared by all three paths, not a measured
ceiling. An omitted `fps` on the
gRPC `StartStreaming` path defaults to `24` on both drivers (`GrpcDriverServer.swift`'s
`defaultStreamingFps`, and the inline `if (request.hasFps()) request.fps else 24` in Android's
`DriverServiceImpl.startStreaming` — both adopting the proto's documented default);
`XCViewHierarchyManager`'s `defaultStreamingFps` of `1` defaults only the legacy WebSocket timer
path. The two are deliberately not aligned — aligning them would change the RPC's behaviour rather
than just guard its arithmetic.

#### Scenario: streaming at 24 fps

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -36,8 +36,8 @@ import kotlinx.coroutines.withTimeoutOrNull
/**
* gRPC service implementation for the Android driver.
*
* This replaces ActionProcessor and handles all incoming RPC calls from the
* TypeScript client (packages/device-node).
* Handles all incoming RPC calls from the TypeScript client
* (packages/device-node).
* Each method corresponds to an action that can be performed on the device.
*/
class DriverServiceImpl : DriverServiceGrpc.DriverServiceImplBase() {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@ import java.util.concurrent.TimeUnit
/**
* gRPC server for the Android driver.
*
* This replaces WebSocketServerImpl. It starts a gRPC server on the specified port
* and handles incoming RPC calls from the TypeScript client (packages/device-node).
* Starts a gRPC server on the specified port and handles incoming RPC calls
* from the TypeScript client (packages/device-node).
*/
class GrpcDriverServer(private val port: Int) {
private var server: Server? = null
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,3 +9,4 @@
{"event":"review","result":"passed","ts":"2026-07-31T02:29:55Z"}
{"action":"enter","driver":"fab-fff","event":"stage-transition","stage":"ship","ts":"2026-07-31T02:33:34Z"}
{"action":"enter","driver":"git-pr","event":"stage-transition","stage":"review-pr","ts":"2026-07-31T02:35:20Z"}
{"event":"review","result":"passed","ts":"2026-07-31T12:20:33Z"}
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ progress:
review: done
hydrate: done
ship: done
review-pr: active
review-pr: done
plan:
generated: true
task_count: 7
Expand All @@ -34,7 +34,7 @@ stage_metrics:
review: {started_at: "2026-07-31T02:19:15Z", driver: fab-fff, iterations: 1, completed_at: "2026-07-31T02:29:55Z"}
hydrate: {started_at: "2026-07-31T02:29:55Z", driver: fab-fff, iterations: 1, completed_at: "2026-07-31T02:33:34Z"}
ship: {started_at: "2026-07-31T02:33:34Z", driver: fab-fff, iterations: 1, completed_at: "2026-07-31T02:35:20Z"}
review-pr: {started_at: "2026-07-31T02:35:20Z", driver: git-pr, iterations: 1}
review-pr: {started_at: "2026-07-31T02:35:20Z", driver: git-pr, iterations: 1, completed_at: "2026-07-31T12:20:33Z"}
prs:
- https://github.com/droid-ash/finalrun-agent/pull/170
change_type_source: explicit
Expand All @@ -50,4 +50,4 @@ true_impact:
computed_at_stage: ship
summary: 'Deleted 1,864 lines of dead code: TestActions.kt (whole file), XCTestManager.swift commented-out code blocks, 30 unreferenced constants.ts exports, and isInteractive/resolveCliCacheRoot in packages/cli'
# true_impact: lazily created on first stage-finish that computes it (no placeholder here).
last_updated: 2026-07-31T02:35:20Z
last_updated: 2026-07-31T12:20:33Z
Loading
Loading