feat!: add PE-A evidence retrieval and agent evaluation

.github/workflows/desktop.yml

@@ -183,10 +183,10 @@ jobs:
         run: npm run check
         working-directory: desktop
 
-      - name: Verify generated desktop contracts and notices
+      - name: Verify generated desktop inputs and notices
         run: |
+          npm run model-catalog:generate
           cargo fetch --manifest-path src-tauri/Cargo.toml --locked
-          npm run model-catalog:check
           npm run notices:check
         working-directory: desktop
 

.gitignore

@@ -42,6 +42,7 @@ pnpm-debug.log*
 /target
 /desktop/node_modules/
 /desktop/dist/
+/desktop/generated/
 /premiere/dist/
 /premiere/ccx/
 /premiere/coverage/

AGENTS.md

@@ -12,7 +12,13 @@ share the same application contracts. Read
 - Keep capability-specific models, schemas, dependencies, indexing, and search
   logic under `src/vidxp/capabilities/`.
 - Do not commit generated environments, model weights, media, indexes, build
-  outputs, or local data.
+  outputs, local databases, or other bulky or machine-specific data.
+- You may commit lightweight, sanitized benchmark result artifacts when they
+  support the change or a documented research claim and remain useful from a
+  different checkout. Reference each retained artifact from the owning
+  benchmark documentation and include only the configuration, provenance,
+  metrics, and traces needed to audit it. Contributors may propose these
+  artifacts; maintainers decide whether they remain before merge.
 - Follow the Conventional Commit and release-note rules in the contributing
   guide. State explicitly when a change is internal-only.
 

INSTALLATION_GUIDE.md

@@ -245,6 +245,22 @@ vidxp search speech "the bread just came out of the oven"
 Add `--media-id <media-id>` to a search command to restrict results to one
 video. Without it, VidXP searches all indexed videos in the active repository.
 
+Sound indexing defaults to one ten-second inference section per batch, two
+seconds of overlap, and ten-second search results. Override a value only when
+running a declared experiment, for example:
+
+```bash
+vidxp index create <media-id> --modality sound \
+  --option sound.inference_overlap_seconds=5
+```
+
+The corresponding option names are `batch_size`,
+`inference_window_seconds`, `inference_overlap_seconds`, and
+`evidence_window_seconds`. The overlap must be shorter than the inference
+window. The inference window bounds one model call, not the media duration;
+long videos are processed as successive sections. These settings change the
+index profile and require the affected media to be indexed again.
+
 ### Start an installed interface
 
 | Interface | Command |
@@ -316,26 +332,33 @@ and sharing behavior.
 ## Optional local grounded answers
 
 VidXP search does not require a language model. To let CLI, HTTP, or MCP
-queries plan searches and draft grounded answers locally, enable **Local
-grounded answers** in VidXP Desktop setup. Desktop checks for a compatible
-loopback Ollama service and then for an existing Ollama executable. When neither
-is available on Windows x86-64 or macOS Apple Silicon, Desktop asks before
-downloading a pinned, checksum-verified headless runtime into VidXP's private
-data. It does not install the Ollama desktop app. Linux setup links to Ollama's
-official installation instructions instead of running a privileged script.
+queries plan searches and draft grounded answers locally, run:
+
+```bash
+vidxp local-answers prepare
+vidxp local-answers status
+```
+
+The recommended `local-worker` installation already includes the required
+client. Custom package installations must include the `slm` extra.
+
+Preparation reuses a healthy self-hosted Ollama service or an existing Ollama
+executable. When neither is available on Windows x86-64 or macOS Apple Silicon,
+VidXP discloses the download sizes and asks before installing its pinned,
+checksum-verified headless runtime. It does not install the Ollama desktop app.
+Linux requires an existing Ollama installation.
 
 This optional feature follows Ollama's platform floor: Windows 10 22H2 or
 newer, or macOS 14 or newer. VidXP Desktop itself can still run without local
 grounded answers on older supported systems.
 
-The model is an additional approximately 3.4 GB download. When Desktop must
-provide the headless runtime, that download is up to approximately 1.36 GiB;
-reusing Ollama avoids it. Local answers have no per-run API charge or numbered
-hosted-model allowance, but they use local storage, memory, compute time, and
-electricity. Desktop configures the private service address for its browser,
-worker, API, Premiere, and generated MCP/Codex setup; there is no URL field to
-fill in. A command-line-only installation remains available for developers and
-custom deployments.
+The model is an additional approximately 3.4 GB download. A managed headless
+runtime can add up to approximately 1.36 GiB; reusing Ollama avoids it. Local
+answers have no per-run API charge or numbered hosted-model allowance, but they
+use local storage, memory, compute time, and electricity. The command saves the
+local endpoint and model selection, so later VidXP CLI, HTTP, and MCP processes
+do not need shell exports. Desktop setup invokes the same preparation operation
+inside its managed runtime and carries the settings into Desktop-owned services.
 
 The complete setup and its current evidence limitations are documented under
 [Enable local grounded answers](docs/local-api.md#enable-local-grounded-answers).
@@ -427,9 +450,12 @@ vidxp doctor
 ```
 
 When an upgrade changes a search model or index format, existing videos may need
-to be indexed again. VidXP reports this instead of silently replacing a working
-index. Prepare the required models, re-index the affected videos, and keep the
-old repository until you have checked the replacement results.
+to be indexed again. Index schema 8 replaces FineLAP sound records with
+PE-A-Frame records and changes vector collections to inner-product ranking, so
+repositories from schema 7 must be rebuilt. VidXP reports the incompatibility
+instead of silently replacing a working index. Prepare the required models,
+re-index the affected videos, and keep the old repository until you have checked
+the replacement results.
 
 The current public capability names are `scene`, `action`, `sound`, `speech`,
 and `actor`. VidXP does not translate removed capability names.

README.md

@@ -170,6 +170,10 @@ questions, and return inspectable evidence such as boards, frames, and clips.
 A local client can start VidXP as a program on the same computer. A hosted
 client connects to a deployed VidXP server.
 
+The goal is to give the agent useful eyes and ears without sending the whole
+video through its context. VidXP narrows the library to timestamped evidence;
+the agent inspects that evidence and decides what it means.
+
 ### Codex plugin and skills
 
 VidXP is distributed as a Codex plugin through a Git marketplace hosted in

benchmarks/codex-mcp/package.json

@@ -2,18 +2,23 @@
   "name": "vidxp-codex-mcp-eval",
   "private": true,
   "version": "0.0.0",
-  "description": "Paired Codex evaluation with and without the local VidXP MCP server",
+  "description": "VidXP agent-ablation and local-agent evaluation",
   "engines": {
     "node": ">=22.22.0"
   },
   "scripts": {
     "setup": "node scripts/setup.mjs",
-    "test:setup": "node --test scripts/setup.test.mjs",
+    "test:setup": "node --no-warnings --test scripts/setup.test.mjs scripts/report.test.mjs",
     "promptfoo": "node --env-file=.env node_modules/promptfoo/dist/src/entrypoint.js",
     "check": "node scripts/require-node.mjs && node --env-file-if-exists=.env node_modules/promptfoo/dist/src/entrypoint.js validate -c promptfooconfig.yaml",
     "preflight": "node --env-file=.env scripts/preflight.mjs",
-    "eval:smoke": "npm run preflight && npm run promptfoo -- eval -c promptfooconfig.yaml --filter-first-n 2 --repeat 1 --no-cache --no-share",
-    "eval:pilot": "npm run preflight && npm run promptfoo -- eval -c promptfooconfig.yaml --repeat 3 --no-cache --no-share",
+    "eval:smoke": "node scripts/require-node.mjs && node --env-file=.env --no-warnings scripts/run-eval.mjs smoke",
+    "eval:pilot": "node scripts/require-node.mjs && node --env-file=.env --no-warnings scripts/run-eval.mjs pilot",
+    "eval:vidxp": "node scripts/require-node.mjs && node --env-file=.env --no-warnings scripts/run-eval.mjs pilot vidxp-on",
+    "eval:slm:smoke": "node scripts/require-node.mjs && node --env-file=.env --no-warnings scripts/run-eval.mjs smoke local-slm,local-slm-planner",
+    "eval:slm": "node scripts/require-node.mjs && node --env-file=.env --no-warnings scripts/run-eval.mjs pilot local-slm,local-slm-planner",
+    "export": "node --env-file=.env --no-warnings scripts/export-eval.mjs",
+    "report": "node --env-file-if-exists=.env --no-warnings scripts/report.mjs",
     "view": "npm run promptfoo -- view"
   },
   "devDependencies": {

benchmarks/codex-mcp/promptfooconfig.yaml

@@ -1,5 +1,8 @@
 # yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
-description: VidXP Codex MCP-on versus MCP-off temporal evidence evaluation
+description: VidXP agent ablation and local-planning temporal evidence evaluation
+
+extensions:
+  - file://scripts/reset-workspace.mjs:beforeEach
 
 prompts:
   - id: video-evidence-task
@@ -8,16 +11,14 @@ prompts:
 
 providers:
   - id: openai:codex-sdk
-    label: codex-vidxp-mcp
-    config:
+    label: codex-vidxp
+    config: &vidxp_provider
       model: "{{ env.VIDXP_EVAL_MODEL | default('gpt-5.6-sol') }}"
       model_reasoning_effort: "{{ env.VIDXP_EVAL_REASONING | default('medium') }}"
       maxRetries: 0
-      working_dir: "{{ env.VIDXP_EVAL_WORKSPACE }}"
+      working_dir: "{{ env.VIDXP_EVAL_VIDXP_ON_WORKSPACE }}"
       skip_git_repo_check: true
-      sandbox_mode: read-only
       approval_policy: never
-      network_access_enabled: false
       web_search_mode: disabled
       persist_threads: false
       enable_streaming: true
@@ -27,65 +28,65 @@ providers:
         required:
           - video_id
           - answer
-          - start_seconds
-          - end_seconds
-          - modalities
-          - evidence
+          - source_job_id
+          - candidates
         properties:
           video_id:
             type: string
           answer:
             type: string
-          start_seconds:
-            type:
-              - number
-              - "null"
-          end_seconds:
+          source_job_id:
             type:
-              - number
+              - string
               - "null"
-          modalities:
-            type: array
-            uniqueItems: true
-            items:
-              type: string
-              enum:
-                - scene
-                - action
-                - sound
-                - speech
-          evidence:
+          candidates:
             type: array
+            minItems: 0
+            maxItems: 3
             items:
               type: object
               additionalProperties: false
               required:
                 - start_seconds
                 - end_seconds
-                - modality
+                - modalities
                 - description
+                - evidence_ids
               properties:
                 start_seconds:
                   type: number
                 end_seconds:
                   type: number
-                modality:
-                  type: string
-                  enum:
-                    - scene
-                    - action
-                    - sound
-                    - speech
+                modalities:
+                  type: array
+                  items:
+                    type: string
+                    enum:
+                      - scene
+                      - action
+                      - sound
+                      - speech
                 description:
                   type: string
+                evidence_ids:
+                  type: array
+                  items:
+                    type: string
       cli_env:
-        CODEX_HOME: "{{ env.VIDXP_EVAL_CODEX_HOME }}"
+        CODEX_HOME: "{{ env.VIDXP_EVAL_VIDXP_ON_CODEX_HOME }}"
+        HOME: "{{ env.VIDXP_EVAL_VIDXP_ON_WORKSPACE }}"
+        TMPDIR: "{{ env.VIDXP_EVAL_VIDXP_ON_WORKSPACE }}/tmp"
       cli_config:
         features:
           multi_agent: false
         mcp_servers:
           vidxp:
             command: "{{ env.VIDXP_MCP_COMMAND }}"
+            env:
+              VIDXP_MODEL_CACHE: "{{ env.VIDXP_MODEL_CACHE }}"
+              VIDXP_ALLOW_MODEL_DOWNLOADS: "false"
+              VIDXP_SLM_BASE_URL: ""
+              VIDXP_SLM_MODEL: ""
             args:
               - --repository
               - "{{ env.VIDXP_EVAL_REPOSITORY | default('default') }}"
@@ -97,33 +98,60 @@ providers:
               - "{{ env.VIDXP_EVAL_DEVICE | default('cpu') }}"
 
   - id: openai:codex-sdk
-    label: codex-no-mcp
+    label: codex-baseline
     config:
-      model: "{{ env.VIDXP_EVAL_MODEL | default('gpt-5.6-sol') }}"
-      model_reasoning_effort: "{{ env.VIDXP_EVAL_REASONING | default('medium') }}"
-      maxRetries: 0
-      working_dir: "{{ env.VIDXP_EVAL_WORKSPACE }}"
-      skip_git_repo_check: true
-      sandbox_mode: read-only
-      approval_policy: never
-      network_access_enabled: false
-      web_search_mode: disabled
-      persist_threads: false
-      enable_streaming: true
-      output_schema: *result_schema
+      <<: *vidxp_provider
+      working_dir: "{{ env.VIDXP_EVAL_VIDXP_OFF_WORKSPACE }}"
+      cli_env:
+        CODEX_HOME: "{{ env.VIDXP_EVAL_VIDXP_OFF_CODEX_HOME }}"
+        HOME: "{{ env.VIDXP_EVAL_VIDXP_OFF_WORKSPACE }}"
+        TMPDIR: "{{ env.VIDXP_EVAL_VIDXP_OFF_WORKSPACE }}/tmp"
+      cli_config:
+        features:
+          multi_agent: false
+
+  - id: openai:codex-sdk
+    label: codex-clean-user
+    config:
+      <<: *vidxp_provider
+      working_dir: "{{ env.VIDXP_EVAL_CLEAN_USER_WORKSPACE }}"
       cli_env:
-        CODEX_HOME: "{{ env.VIDXP_EVAL_CODEX_HOME }}"
+        CODEX_HOME: "{{ env.VIDXP_EVAL_CLEAN_USER_CODEX_HOME }}"
+        HOME: "{{ env.VIDXP_EVAL_CLEAN_USER_WORKSPACE }}"
+        PATH: "{{ env.VIDXP_EVAL_CLEAN_USER_PATH }}"
+        TMPDIR: "{{ env.VIDXP_EVAL_CLEAN_USER_WORKSPACE }}/tmp"
       cli_config:
         features:
           multi_agent: false
 
+  - id: file://scripts/local_slm_provider.py:call_api
+    label: local-slm
+    config: &local_slm_provider
+      pythonExecutable: "{{ env.PROMPTFOO_PYTHON }}"
+      workers: 1
+      timeout: 900000
+      contextTokens: 64000
+      maxOutputTokens: 32768
+      modelTimeoutSeconds: 180
+      strategy: router
+
+  - id: file://scripts/local_slm_provider.py:call_api
+    label: local-slm-planner
+    config:
+      <<: *local_slm_provider
+      strategy: planner
+
 tests:
   - path: file://../../src/vidxp/benchmarks/agent_ablation_tests.py:generate_tests
     config:
       manifest: tasks/longvale-part9-pilot.json
+      machine_id: "{{ env.VIDXP_EVAL_MACHINE_ID }}"
       providers:
-        mcp_on: codex-vidxp-mcp
-        mcp_off: codex-no-mcp
+        vidxp_on: codex-vidxp
+        vidxp_off: codex-baseline
+        clean_user: codex-clean-user
+        local_slm: local-slm
+        local_slm_planner: local-slm-planner
 
 evaluateOptions:
   cache: false

benchmarks/codex-mcp/prompts/local-slm-planner.txt

@@ -0,0 +1,20 @@
+Plan one ranked VidXP search for the supplied event.
+
+Return:
+
+- query: a concise search query that preserves every detail needed to
+  distinguish the event; rephrase only when it makes those details clearer
+- modalities: every relevant indexed evidence type and no unrelated type
+- candidate_top_k: the number of hits VidXP should retrieve from each selected
+  modality before fusion, from 3 through 100; use a deeper pool when the event
+  is ambiguous or likely to repeat
+
+The evidence types are:
+
+- scene: visible objects, setting, appearance, or visual state
+- action: visible movement, activity, or change over time
+- sound: non-speech audio, including environmental and mechanical sounds
+- speech: spoken words or dialogue
+
+Do not predict timestamps, answer the request, or choose the final result count.
+The harness always returns VidXP's top three fused evidence results.

benchmarks/codex-mcp/prompts/local-slm-router.txt

@@ -0,0 +1,10 @@
+Choose which indexed VidXP evidence types are relevant to locating the supplied
+event. Return every relevant type and no unrelated type.
+
+- scene: visible objects, setting, appearance, or visual state
+- action: visible movement, activity, or change over time
+- sound: non-speech audio, including environmental and mechanical sounds
+- speech: spoken words or dialogue
+
+Do not locate the event, predict timestamps, rewrite the query, or summarize
+results. Your only task is modality selection.