Integrate dev workflow hardening and runtime testability improvements

.agent/PLANS.md

@@ -0,0 +1,58 @@
+# Planning Convention
+
+Use this file when the work is large enough that an agent should externalize its plan before or during implementation.
+
+## When To Write A Plan
+
+Write a plan when the task:
+
+- spans multiple packages or mixes code, docs, and workflow changes,
+- changes repository behavior, contributor workflow, or validation expectations,
+- updates active interfaces, launch flows, or runtime assumptions,
+- or is likely to require follow-up work after the current change.
+
+Skip formal planning only for narrow, obviously local edits.
+
+## Plan Template
+
+Keep plans short and operational:
+
+```md
+# Goal
+- One or two sentences on the desired outcome.
+
+# Boundaries
+- What is in scope.
+- What is explicitly out of scope.
+
+# Evidence
+- Files, docs, tests, configs, or upstream references that define the contract.
+
+# Steps
+- Ordered implementation steps.
+
+# Validation
+- Commands to run.
+- Evidence that will prove the change works in this repository.
+
+# Risks / Follow-ups
+- Remaining uncertainty, deferred scope, or issue candidates.
+```
+
+## Required Behaviors
+
+- Restate the repository goal in PAF terms, not generic assistant language.
+- Separate observed evidence from assumptions.
+- Prefer committed scripts, VS Code tasks, and documented workflows over ad-hoc commands.
+- For non-trivial development work, state the proof obligation before implementation.
+- Prefer the consolidation order from `doc/dev_talks/paf25/future_work.md`: documentation and interfaces first, then tests and CI, then subsystem fixes.
+- If scope expands, capture a follow-up in `doc/reasoning/` or the issue tracker instead of silently broadening the change.
+
+## Review Expectations
+
+A good plan makes it easy for a reviewer to answer:
+
+- what changed,
+- why that scope is correct,
+- how it was validated,
+- and what risk remains.

.devcontainer/devcontainer.json

@@ -0,0 +1,46 @@
+{
+  "name": "PAF Agent Dev (CUDA)",
+  "dockerComposeFile": [
+    "../build/docker-compose.dev.cuda.yml"
+  ],
+  "service": "agent-dev",
+  "runServices": [
+    "agent-dev"
+  ],
+  "workspaceFolder": "/workspace",
+  "containerEnv": {
+    "PAF_LAUNCH_CONTAINER_VSCODE": "0"
+  },
+  "initializeCommand": "bash ${localWorkspaceFolder}/scripts/update-dotenv.sh",
+  "shutdownAction": "stopCompose",
+  "overrideCommand": false,
+  "postStartCommand": "bash -lc 'source /internal_workspace/dev.bashrc >/dev/null 2>&1 && ros2 --version'",
+  "customizations": {
+    "vscode": {
+      "extensions": [
+        "davidanson.vscode-markdownlint",
+        "github.copilot",
+        "github.copilot-chat",
+        "github.vscode-pull-request-github",
+        "ms-azuretools.vscode-docker",
+        "ms-vscode.cmake-tools",
+        "ms-vscode-remote.remote-containers",
+        "ms-python.python",
+        "ms-python.vscode-pylance",
+        "charliermarsh.ruff",
+        "editorconfig.editorconfig",
+        "augustocdias.tasks-shell-input",
+        "ranch-hand-robotics.rde-pack",
+        "ktnrg45.vscode-cython",
+        "timonwong.shellcheck",
+        "phil294.git-log--graph",
+        "richardkotze.git-mob",
+        "bierner.markdown-mermaid",
+        "valentjn.vscode-ltex",
+        "vscode-icons-team.vscode-icons",
+        "yzhang.markdown-all-in-one",
+        "njpwerner.autodocstring"
+      ]
+    }
+  }
+}

.github/instructions/docs-and-reasoning.instructions.md

@@ -0,0 +1,15 @@
+---
+description: "Use when updating documentation, architecture notes, contributor guidance, or saved reasoning. Covers doc sync with code, canonical docs vs reasoning notes, and the doc/reasoning workflow."
+name: "Docs And Reasoning"
+applyTo: "doc/**/*.md,README.md,agents.md"
+---
+# Docs And Reasoning
+
+- Active documentation must match the current implementation. If code and docs disagree, either fix the doc in the same change or call out the gap explicitly.
+- Keep canonical behavior and interface docs in their domain folders under `doc/` or the package docs.
+  Use `doc/dev/progress/` for timestamped progress and handoff notes that capture the current chat state, and use `doc/reasoning/` for deeper analysis notes, comparisons, migration thoughts, and development output that should not become the source of truth.
+- When saving a progress or handoff note, include the timestamp, motivating task, current chat state, source files or repositories inspected, validation status, remaining blockers, and recommended follow-ups.
+- When saving a reasoning note, include the motivating task, the source files or repositories inspected, the main conclusion, and the remaining follow-ups.
+- Link new documentation from `doc/README.md` or the most relevant existing index page so it stays discoverable.
+- Add documentation hints for new progress notes by updating `doc/dev/README.md`, `doc/README.md`, or the closest relevant module or development doc.
+- Prefer short, actionable Markdown over long narrative dumps. Remove or update stale statements instead of piling on contradictory notes.

.github/instructions/repo-consolidation.instructions.md

@@ -0,0 +1,12 @@
+---
+description: "Use when doing repository-wide cleanup, development workflow changes, cross-package fixes, or quality hardening. Covers consolidation-before-expansion, interface discipline, and proof-first validation for PAF."
+name: "Repository Consolidation"
+---
+# Repository Consolidation
+
+- Use `doc/dev_talks/paf25/future_work.md` and `doc/dev_talks/paf25/improvements_assessment.md` as the direction-setting documents for repo-wide changes.
+- Prefer consolidation work before new feature breadth: sync active docs, clarify interfaces, strengthen tests and CI, and reduce friction in development workflows.
+- When a change touches more than one package or mixes workflow plus runtime behavior, make the controlling contract explicit: name the package boundary, topic/message, task, or doc that defines the expected behavior.
+- Keep changes small and reversible. If the best answer is larger than one reviewable slice, implement the smallest slice now and record the rest as follow-up work.
+- State proof before implementation. Typical proof in this repo is one of: Ruff lint/format, host smoke tests, ROS-backed unit tests, route-level validation, or docs/link verification.
+- If a change updates an interface or development rule, update the canonical documentation in the same change or record the gap in `doc/reasoning/`.

.github/pull_request_template.md

@@ -21,7 +21,19 @@ e.g. is old functionality not usable anymore
 
 Which files functionalities are most important in this PR. On which part should the reviewer be focussed on?
 
-# Checklist:
+## Assumptions and constraints
+
+List assumptions, environment constraints, and any scope limits used for this change.
+
+## Validation summary
+
+List commands executed and outcomes (lint, format, tests, simulation checks, etc.).
+
+## Known gaps / follow-ups
+
+List remaining risks or follow-up work that is intentionally out of scope.
+
+## Checklist
 
 - [ ] My code follows the style guidelines of this project
 - [ ] I have performed a self-review of my own code
@@ -30,4 +42,5 @@ Which files functionalities are most important in this PR. On which part should
 - [ ] My changes generate no new warnings
 - [ ] I have added tests that prove my fix is effective or that my feature works (might be obsolete with CI later on)
 - [ ] New and existing unit tests pass locally with my changes (might be obsolete with CI later on)
-
+- [ ] I documented assumptions, validation steps, and known gaps in this PR
+- [ ] I ran dependency validation (`dep.check` or `scripts/dependency-doctor.sh`) when touching dependencies

.github/workflows/dependency-doctor.yml

@@ -0,0 +1,27 @@
+name: Dependency doctor
+
+on:
+  pull_request:
+    branches:
+      - main
+  push:
+    branches:
+      - main
+
+jobs:
+  dependency-doctor:
+    name: Validate dependency manifests
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - name: Verify pinned and consistent requirements
+        run: bash scripts/dependency-doctor.sh
+      - name: Install infra dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -r code/requirements_infrastructure.txt
+      - name: Validate Python package dependency graph
+        run: python -m pip check

.github/workflows/drive.yml

@@ -16,13 +16,13 @@ jobs:
     if: ${{ github.event.workflow_run.conclusion == 'success' }}
     steps:
       - name: Checkout repository
-        uses: actions/checkout@v3
+        uses: actions/checkout@v5
       - name: Print environment variables (DEBUG)
         run: |
           echo "AGENT_VERSION=${AGENT_VERSION}"
           echo "COMPOSE_FILE=${COMPOSE_FILE}"
       - name: Download artifact
-        uses: actions/github-script@v6
+        uses: actions/github-script@v8
         with:
           script: |
             let allArtifacts = await github.rest.actions.listWorkflowRunArtifacts({
@@ -49,27 +49,27 @@ jobs:
         run: unzip artifact.zip
       - name: Return artifact JSON
         id: return-artifact-json
-        uses: actions/github-script@v6
+        uses: actions/github-script@v8
         with:
           script: |
             let fs = require('fs');
             let data = JSON.parse(fs.readFileSync(`${process.env.GITHUB_WORKSPACE}/artifact.json`));
             return data;
 
-      - name: Run docker-compose
+      - name: Run Docker Compose
         run: |
           xhost +local:
           USERNAME=$(whoami) USER_UID=$(id -u) USER_GID=$(id -g) RENDER_OFFSCREEN=-RenderOffScreen docker compose up --quiet-pull --exit-code-from agent
       - name: Copy results
         run: docker compose cp agent:/tmp/simulation_results.json .
-      - name: Stop docker-compose
+      - name: Stop Docker Compose
         # always run this step, to clean up even on error
         if: always()
         run: docker compose down -v
       # add rendered JSON as comment to the pull request
       - name: Create simulation results table
         id: simulation-results
-        uses: actions/github-script@v6
+        uses: actions/github-script@v8
         with:
           script: |
             const fs = require('fs');
@@ -94,7 +94,7 @@ jobs:
           echo "${{ steps.simulation-results.outputs.result }}"
       - name: Add simulation results as comment
         if: ${{ steps.return-artifact-json.outputs.result.is_pr }}
-        uses: actions/github-script@v6
+        uses: actions/github-script@v8
         with:
           github-token: ${{ secrets.GITHUB_TOKEN }}
           # this script reads the simulation_results.json and creates a comment on the pull request with the results.

.github/workflows/markdownlint.yml

@@ -14,7 +14,7 @@ jobs:
         runs-on: ubuntu-latest
         steps:
           - name: Check out the repo
-            uses: actions/checkout@v2
+            uses: actions/checkout@v5
           # Execute the markdown linter
           - name: Run the markdown linter
             uses: addnab/docker-run-action@v3

.github/workflows/ros-unit-tests.yml

@@ -0,0 +1,40 @@
+name: ROS backed unit tests container
+
+on:
+    pull_request:
+        branches:
+            - main
+    push:
+        branches:
+            - main
+
+jobs:
+    ros_unit_tests:
+        name: Run ROS-backed unit tests in agent container
+        runs-on:
+            - self-hosted
+            - build
+        steps:
+            - uses: actions/checkout@v4
+            - name: Set test image tag
+              run: echo "TEST_IMAGE=paf-agent-deploy-test:${GITHUB_SHA}" >> "$GITHUB_ENV"
+            - name: Build agent-deploy test image
+              run: |
+                  docker build \
+                    --target agent-deploy \
+                    --file build/docker/agent-ros2/Dockerfile \
+                    --build-arg PAF_USERNAME=paf \
+                    --build-arg PAF_UID=1000 \
+                    --build-arg PAF_GID=1000 \
+                    --build-arg BASE_FLAVOUR=cuda \
+                    --tag "${TEST_IMAGE}" \
+                    .
+            - name: Run ROS-backed unit tests
+              run: |
+                  docker run --rm \
+                    --entrypoint bash \
+                    "${TEST_IMAGE}" \
+                    -lc 'source /internal_workspace/deploy.bashrc && PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 python3 -m pytest /workspace/code/perception/tests /workspace/code/mapping/test /workspace/code/planning/test -m unit'
+            - name: Cleanup test image
+              if: always()
+              run: docker image rm -f "${TEST_IMAGE}" || true

.github/workflows/ruff-strict-rollout.yml

@@ -0,0 +1,32 @@
+name: "Ruff strict rollout"
+
+on:
+    pull_request:
+        branches:
+            - main
+    push:
+        branches:
+            - main
+
+jobs:
+    ruff-strict-starter:
+        name: Lint paf_common with Ruff strict
+        runs-on: ubuntu-latest
+        steps:
+            - uses: actions/checkout@v4
+            - name: Load Ruff version
+              id: ruff-version
+              run: |
+                  VERSION_LINE=$(grep -E '^RUFF_VERSION=' build/pins/ruff.env || true)
+                  VERSION=${VERSION_LINE#RUFF_VERSION=}
+                  if [ -z "${VERSION}" ]; then
+                    echo "Ruff version missing in build/pins/ruff.env" >&2
+                    exit 1
+                  fi
+                  echo "version=${VERSION}" >> "$GITHUB_OUTPUT"
+            - name: Install Ruff
+              uses: astral-sh/ruff-action@v3
+              with:
+                  version: ${{ steps.ruff-version.outputs.version }}
+            - name: Lint starter package with strict profile
+              run: ruff check code/paf_common --config ruff-strict.toml

.github/workflows/unit-tests.yml

@@ -0,0 +1,21 @@
+name: Host smoke tests
+
+on:
+  pull_request:
+    branches:
+      - main
+  push:
+    branches:
+      - main
+
+jobs:
+  host-smoke-tests:
+    name: Run host smoke tests
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - name: Run host smoke tests
+        run: bash scripts/run-host-smoke-tests.sh

.gitignore

@@ -1,6 +1,8 @@
 .DS_Store
 .idea
 .gitconfig
+.venv/
+.venv-host/
 
 /volumes
 
@@ -22,3 +24,9 @@ __pycache__/
 # Auto downloaded models
 code/*.pt
 /*.pt
+
+# VS Code transient browse database files
+.vscode/browse.vc.db*
+
+tmp/*
+!tmp/.gitignore

.pre-commit-config.yaml

@@ -0,0 +1,13 @@
+repos:
+  - repo: https://github.com/astral-sh/ruff-pre-commit
+    rev: v0.14.8
+    hooks:
+      - id: ruff
+        args: ["--fix"]
+      - id: ruff-format
+
+  - repo: https://github.com/igorshubovych/markdownlint-cli
+    rev: v0.45.0
+    hooks:
+      - id: markdownlint
+        args: ["--config", ".markdownlint.yaml"]