Personal Phone Agent

Personal Phone Agent is a Raspberry Pi voice-agent prototype for bounded customer-service follow-ups. The owner supplies a private call brief with the objective and constraints, including allowed disclosures and escalation rules. Authentication, financial commitments, and contracts stay with a human.

The project tests whether this setup can handle the waiting, menu navigation, and routine conversation that make administrative calls expensive in time without giving the agent open-ended authority.

The Raspberry Pi is the telephone and audio gateway: it controls an Android Pixel over wireless ADB and connects to it through Bluetooth HFP/SCO for bidirectional call audio. This prototype instruments the full path—from hardware and network transport to streaming speech and dialogue providers—to test whether that delegation can be useful, credible, and responsible in practice.

What works today

• Raspberry Pi deployment support: wireless ADB health checks, Bluetooth HFP/SCO audio routing, a local observatory, and passive hardware monitoring.
• A local browser conversation loop: microphone input, streaming transcription, turn control, an LLM, speech synthesis, and barge-in.
• Deterministic simulations for calls, IVRs, DTMF, failures, and human handoffs, with no phone call, network request, or audio recording.
• Replaceable adapters for ADB, PipeWire, STT, LLM, and TTS. Cartesia and ElevenLabs can output 16 kHz PCM for the phone audio path.
• A local observatory with a timeline, metrics, SSE events, a conversation view, and passive hardware monitoring.
• Tested safety boundaries: explicit authorization, bounded objectives, escalation for authentication and commitments, and redaction of sensitive data in persistent events.

This is still a prototype: end-to-end duplex Bluetooth SCO integration on a real call needs repeatable validation.

Architecture

                 Raspberry Pi 5 (deployed runtime)
    ┌─────────────────────────────────────────────────────┐
    │ Conversational core: STT -> turns -> LLM -> TTS     │
    │ Safety policy, observatory, and audio RX / TX buses │
    └───────────────┬───────────────────┬─────────────────┘
                    │                   │
            Wireless ADB            Bluetooth HFP / SCO
                    │                   │
                    └─────── Android Pixel ───────┘
                                  |
                           cellular call audio

For local development, a browser microphone and speakers use the same conversational components without requiring phone hardware or its Bluetooth and cellular connections.

The core depends on ports instead of ADB, PipeWire, or any AI provider. That makes critical behavior testable in simulation before a phone integration is attempted. See the runtime architecture and observability model.

Run locally

Prerequisites: Python 3.11+ and PowerShell. STT, LLM, and TTS providers are optional for simulation; a local voice conversation requires their credentials in Git-ignored files only.

cd runtime
.\scripts\setup-local.ps1
.\scripts\start-local.ps1

The dashboard opens at http://127.0.0.1:8787. It uses only the workstation microphone and speakers: it does not place a call, contact the Raspberry Pi, or record audio.

To run the scenarios without an external provider:

cd runtime
$env:PYTHONPATH = "$PWD\src"
python -m phone_agent.cli simulate
python -m unittest discover -s tests -v

Deployment and supervision procedures are in the runtime runbook and Raspberry Pi setup.

Private configuration

Never commit private runtime data. The root .gitignore excludes these expected locations:

• pi-setup/connection.json and pi-setup/connection.local.json;
• concrete call briefs in briefs/;
• runtime/deploy/*.env files containing provider credentials;
• logs, recordings, and local Python build artifacts.

Copy *.example.* files to their local equivalent and fill in values outside Git. Templates and fictional scenarios remain versioned to document the configuration contract.

Safety and limitations

• A real call requires explicit authorization and an exact destination.
• Authentication requests, financial actions, signatures, and commitments must be handed over to a human.
• The agent must be transparent about its role when asked; this project must not be used to impersonate an identity or voice.
• Review applicable consent rules before capturing or retaining audio.

For the runtime safeguards, see the architecture and observability documentation.

Repository layout

briefs/          Call-brief templates and documentation
pi-setup/        Controlled Raspberry Pi setup
prompt-library/  Persona, prompt templates, and manifests
runtime/         Python core, tests, scenarios, and operations

Status

Personal prototype under active development. The isolated components and local mode are testable; the operational reliability of a real telephone call has not yet been established.