Visual Context Router

Stop sending full screenshots to agents.

Visual Context Router is a token throttle for screen-reading AI agents. It lets Codex and other agents inspect screens through structured wireframes, change detection, and ROI crops before falling back to full screenshots.

The goal is simple: send pixels only when pixels are the best evidence.

Demo

On the included sample UI screen:

Input path	Full image tokens	Routed text tokens	Saved	Savings
examples/sample-screen-after.png	768	65	703	91.5%

vcr route examples/sample-screen-after.png --previous examples/sample-screen.png --json

{
  "strategy": "wireframe_plus_roi",
  "include_full_image": false,
  "include_wireframe": true,
  "token_estimate": {
    "full_image_tokens": 768,
    "routed_text_tokens": 65,
    "saved_tokens": 703,
    "savings_ratio": 0.915
  }
}

Reality Check

Visual Context Router is a callable routing tool, not a global Codex screenshot interceptor.

• It can reduce token use when Codex calls vcr_capture_route, vcr_route, or vcr_observe.
• It cannot force Codex to use those tools for every built-in screenshot request.
• After installing the plugin, start a new Codex thread so the MCP tools are loaded.
• If the tools are not visible, run python scripts/doctor.py to check the local install.
• For repeated UI work, use vcr_watch so each observation is compared with the previous screen.

Why This Exists

Vision-capable agents often waste tokens on:

• Unchanged screens.
• Full screenshots when only a dialog or button changed.
• Re-reading UI text that could be represented as structured state.
• Long screen recordings where most frames are redundant.

Visual Context Router sits before the model and decides what to forward:

screen / frame
  -> visual diff
  -> optional OCR + UI element extraction
  -> compact wireframe prompt
  -> optional ROI crops
  -> model

Features

• Screenshot diffing with MSE and normalized change score.
• Compact wireframe prompt generation.
• Region-of-interest cropping by normalized coordinates.
• Heuristic UI element detection with optional OpenCV enhancement.
• Optional OCR with pytesseract.
• JSON observations for easy agent integration.
• Routing decisions: skip, wireframe_only, wireframe_plus_roi, or full_image.
• Codex plugin metadata and skill instructions.

Install

For local CLI/SDK use:

pip install -e .

Optional OCR and OpenCV support:

pip install -e ".[all]"

Install As A Codex Plugin

Clone this repository, then run:

python scripts/install_codex_plugin.py

The installer copies the plugin into your local Codex plugin directory, writes the personal marketplace entry, installs Python dependencies, and tries to run:

codex plugin add visual-context-router@personal

After installing, start a new Codex thread and ask:

Use Visual Context Router to inspect my screen with vcr_capture_route.

See docs/codex-install.md for manual installation and troubleshooting. See docs/prompts.md for copy-paste Codex prompts. See docs/benchmarks.md for reproducible token-saving examples.

Reproduce the included benchmark:

python scripts/benchmark_examples.py

Check the local install:

python scripts/doctor.py

Quick Start

Analyze a screenshot:

vcr observe examples/sample-screen.png

Compare two screenshots and emit JSON:

vcr observe examples/sample-screen-after.png --previous examples/sample-screen.png --json

Crop a region of interest using normalized coordinates:

vcr crop examples/sample-screen.png --bbox 0.70,0.05,0.98,0.30 --out roi.png

Estimate visual token savings:

vcr observe examples/sample-screen.png --json --token-budget

Ask the router what an agent should send next:

vcr route examples/sample-screen-after.png --previous examples/sample-screen.png --json

Watch the current screen with saved previous-frame state:

vcr watch --json

Example Output

{
  "changed": true,
  "change_score": 0.1842,
  "image": {
    "width": 1440,
    "height": 900
  },
  "elements": [
    {
      "id": 1,
      "type": "region",
      "text": "",
      "bbox": [0.1208, 0.1889, 0.4125, 0.2667],
      "confidence": 0.58
    }
  ],
  "wireframe": "Screen 1440x900. Changed: yes..."
}

Routing Strategies

Visual Context Router can make an explicit routing decision for the next model call:

Strategy	Meaning
skip	No meaningful visual change. Continue from action log or cached state.
wireframe_only	Send structured text state only.
wireframe_plus_roi	Send structured text plus a few local crops.
full_image	Send the full screenshot because the agent needs global reorientation.

Agent Pattern

Use the package as a router in front of a multimodal model:

from visual_context_router import observe, route_observation

observation = observe(
    image_path="current.png",
    previous_path="previous.png",
    change_threshold=0.003,
)
decision = route_observation(observation)

if decision.strategy == "skip":
    prompt = "No semantic visual change. Continue from action log."
elif decision.include_wireframe:
    prompt = decision.model_payload

Codex Plugin Usage

This repository includes .codex-plugin/plugin.json and a Codex skill under skills/. It also includes an MCP server that exposes tools Codex can call directly after the plugin is installed:

• vcr_capture_route: capture the current screen and return a low-token routing decision.
• vcr_watch: capture the current screen, compare it with the previous saved capture, and return a low-token routing decision.
• vcr_route: route an existing screenshot.
• vcr_observe: return a compact wireframe observation.
• vcr_crop: crop a normalized region of interest.

Example prompt:

Use Visual Context Router to inspect my current screen with vcr_capture_route before deciding whether you need a full screenshot.

For repeated UI tasks:

Use Visual Context Router with vcr_watch before each screen observation. Only ask for a full screenshot if the route strategy is full_image or ambiguous.

Important: the plugin gives Codex callable tools, but it does not globally intercept every built-in screenshot path. Ask Codex to use vcr_capture_route when you want the low-token route. If the current thread cannot see the tool, start a new Codex thread after installation.

Roadmap

• Accessibility tree adapters for browser and desktop automation.
• DOM change ingestion for browser agents.
• Better icon and control classification.
• Broader token savings benchmark suite.
• Model-facing observation cache.
• Automatic ROI crop bundles for model follow-up.

Development

pip install -e ".[dev]"
pytest

Generate demo screenshots:

python scripts/make_demo_images.py

Generate README assets:

python scripts/make_readme_assets.py

GitHub Release Checklist

• Create a repository named visual-context-router.
• Push this folder as the repo root.
• Add screenshots from examples/ to the README or project social preview.
• Tag the first release as v0.1.0.
• Share a short demo showing full_image_tokens vs routed_text_tokens.

Commercial Angle

The open-source project is useful as a developer tool. Commercial value is likely in:

• Enterprise privacy-preserving local deployment.
• Higher-accuracy UI detection adapters.
• Token cost dashboards.
• Agent framework integrations.
• Browser and desktop automation SDKs.

Support

Visual Context Router is open source. If it helps your Codex or AI agent workflow, optional support is welcome:

爱发电支持 / Support on Afdian

License

MIT