OpenMRS LiveKit

Local-first voice AI for OpenMRS encounters.

OpenMRS LiveKit is a hackathon prototype for privacy-preserving clinical translation and encounter draft generation. It runs beside OpenMRS on clinic hardware, routes audio through LiveKit, transcribes each turn locally, redacts PHI-like values, translates clinician-patient speech, and builds clinician-reviewed OpenMRS draft observations.

It is not a medical device, not a diagnosis engine, and not an autonomous scribe. The system never writes clinical data to OpenMRS without clinician review.

What It Does

• Captures a clinical conversation through a local LiveKit room.
• Supports local or provider-backed STT, translation, LLM extraction, and TTS.
• Redacts PHI-like values before transcript persistence or cloud AI calls.
• Translates clinical speech while preserving negation, uncertainty, medications, dosages, and redaction placeholders.
• Extracts candidate OpenMRS facts with evidence, confidence, speaker, timestamp, and review status.
• Generates OpenMRS-style draft encounter payloads from approved facts only.
• Keeps incomplete or low-confidence items in a review queue.

Why This Exists

Many OpenMRS deployments operate in clinics with unreliable internet, limited staffing, and language or health-literacy barriers between clinicians and patients. Cloud-only AI is a poor fit when PHI cannot leave the facility or when connectivity is unreliable.

This project focuses on a narrow, reusable pattern:

LiveKit voice capture
  -> local STT
  -> PHI redaction
  -> clinical translation
  -> evidence-backed facts
  -> clinician review
  -> OpenMRS draft payload

Repository Pairing

This repo contains the LiveKit voice agent and OpenMRS-safe clinical processing primitives.

The companion OpenMRS O3 frontend lives at:

https://github.com/sihsalus/openmrs-esm-livekit

Together they provide the full demo flow: OpenMRS patient chart, LiveKit room launch, local AI workflow, privacy status, and clinician-reviewed draft UI.

Architecture

Clinician / patient audio
          |
          v
Local LiveKit room
          |
          v
Local or provider STT
faster-whisper / OpenAI / Deepgram
          |
          v
De-identification gateway
          |
          +------------------------------+
          |                              |
          v                              v
Patient-facing translation         Clinical fact extraction
configured LLM / rules             configured LLM + tools
          |                              |
          v                              v
Local or provider TTS               EncounterDraft
Piper / OpenAI / other providers    review queue
          |                              |
          v                              v
Patient hears response          OpenMRS draft payload
                                 after clinician approval

OpenMRS Integration

The intended deployment is a standalone local service or container running next to OpenMRS.

OpenMRS interaction is through REST or FHIR endpoints:

• read patient, visit, encounter type, location, provider, form, and concept metadata;
• build local draft encounter and obs payloads;
• submit only clinician-approved observations;
• keep the demo path synthetic and read-only by default.

The current prototype includes an OpenMRS-style payload builder that only emits approved facts and keeps the rest in a review bundle.

Local AI Stack

The project is designed to be offline-capable, but model providers are configuration-driven.

Default development configuration:

• LLM provider: openai
• Foundation model: gpt-4.1-mini through OPENAI_MODEL
• STT provider: deepgram by default, or openai / whisper
• TTS provider: openai by default, or piper / other configured providers

Recommended local-first demo configuration:

• Audio routing: local LiveKit server.
• STT: whisper provider, backed by faster-whisper, with language-specific clinical prompts and VAD filtering.
• TTS: piper for local CPU voice output, selecting a configured English or Spanish voice from room metadata.
• LLM / structured extraction: ollama provider, defaulting to qwen3:8b through OLLAMA_MODEL. A site can switch to another local model, such as a clinically tuned model, by changing OLLAMA_MODEL.
• Structured output: tool calls, schema-shaped draft objects, and local OpenMRS payload generation.

Minimum reliable clinic target:

• modern 8-core CPU;
• 16 GB RAM recommended;
• 20 GB free disk for models and runtime artifacts;
• no GPU required.

A lighter profile can run on an Intel i5-class mini PC or laptop with 8 GB RAM using Vosk, Piper, and a small quantized parser model.

Current Demo Capabilities

• LiveKit agent foundation for OpenMRS encounter workflows.
• Cloud-safe clinical translation prompt with deterministic PHI placeholders.
• Local de-identification for emails, phone numbers, document IDs, dates, UUIDs, and known entities.
• Reviewable clinical facts with confidence, evidence, speaker, and status.
• OpenMRS-style encounter draft payloads generated only from approved facts.
• Transcript persistence disabled by default, with redaction before any optional save.
• Hackathon submission materials in docs/.

Safety Model

• Human-in-the-loop by default.
• No automatic diagnosis.
• No autonomous prescribing or ordering.
• No automatic OpenMRS writes.
• No production PHI in demos.
• Transcript persistence disabled by default.
• Raw transcript storage requires explicit configuration.
• External AI services should receive only synthetic, redacted, or contractually protected data. The target deployment avoids external AI APIs entirely.

See docs/security-model.md for the detailed privacy and production-hardening model.

Repository Layout

src/
  agent.py                 LiveKit agent entrypoint
  session.py               AgentSession lifecycle and prompt setup
  clinical_translation.py  Cloud-safe/local-safe translation prompt helpers
  deidentification.py      Deterministic PHI-like redaction helpers
  clinical_facts.py        Reviewable clinical fact primitives
  openmrs_payload.py       OpenMRS-style draft payload builder
  transcript.py            Optional transcript persistence with redaction
docs/
  assets/
  hackathon-dossier.md
  submission-form-fields.md
  demo-script.md
  security-model.md
  proposal-positioning.md
tests/
  test_backend_client.py
  test_clinical_facts.py
  test_config.py
  test_deidentification.py
  test_inworld_integration.py
  test_local_providers.py
  test_openmrs_payload.py
  test_openmrs_tools.py
  test_providers_mistral.py
  test_session.py

Local Development

python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"
cp .env.example .env
pytest

Useful checks before opening a PR:

ruff check src/ tests/
ruff format --check src/ tests/
mypy src/ --ignore-missing-imports
pytest --tb=short -q --cov=src --cov-report=term-missing

Use demo or test credentials only.

Example Local-Only Configuration

LIVEKIT_URL=ws://localhost:7880
LIVEKIT_API_KEY=devkey
LIVEKIT_API_SECRET=secret

STT_PROVIDER=whisper
TTS_PROVIDER=piper
LLM_PROVIDER=ollama

ENABLE_TRANSCRIPT_SAVE=false
TRANSCRIPT_REDACTION_ENABLED=true
TRANSCRIPT_RAW_STORAGE_ALLOWED=false

The local provider path uses faster-whisper, Piper TTS, and an Ollama-hosted OpenAI-compatible LLM. The model is not hardcoded into the application; it is selected through LLM_PROVIDER and provider-specific model variables.

Foundation Model And Prompt Language

The agent does not require one fixed foundation model.

Current defaults:

LLM_PROVIDER=openai
OPENAI_MODEL=gpt-4.1-mini

For the local/offline hackathon path:

LLM_PROVIDER=ollama
OLLAMA_MODEL=qwen3:8b

The base system prompt is written in Spanish because the current target demo needs Spanish clinical robustness in a Latin American OpenMRS setting. Runtime language defaults are still English unless the room metadata says OpenMRS is using Spanish. Each LiveKit room can carry normalized language metadata:

{
  "doctorLanguage": "es",
  "patientLanguage": "en",
  "agentVoiceLanguage": "es",
  "speakerAttributionMode": "source-role",
  "defaultHumanRole": "doctor",
  "languageMode": "bilingual",
  "agentProviderOverrides": {
    "sttProvider": "deepgram",
    "ttsProvider": "inworld",
    "deepgramModel": "nova-3",
    "deepgramEnableDiarization": true,
    "deepgramUseFlux": false,
    "inworldModel": "inworld-tts-2"
  }
}

The agent uses doctorLanguage for STT language hints and clinician-facing input, patientLanguage for patient-facing translation context, and agentVoiceLanguage for the initial greeting, assistant transcript labels, TTS language hints, and Piper model selection. Local Whisper uses a clinical initial prompt aligned with the configured STT language.

If room metadata is missing, the agent falls back to English for clinician, patient, and agent voice language. A Spanish OpenMRS locale should send doctorLanguage=es, patientLanguage=es, and usually agentVoiceLanguage=es.

Room metadata can also override the STT/TTS provider before a new AgentSession is created. Supported room-scoped overrides are sttProvider=whisper|deepgram, ttsProvider=piper|inworld, deepgramModel, deepgramEnableDiarization, deepgramUseFlux, and inworldModel. Secrets are never read from room metadata; sttProvider=deepgram is ignored unless DEEPGRAM_API_KEY is configured, and ttsProvider=inworld is ignored unless INWORLD_API_KEY and INWORLD_VOICE_ID are configured.

Speaker attribution is explicit. When the STT provider emits speaker_id (for example Deepgram Nova with DEEPGRAM_ENABLE_DIARIZATION=true), the agent maps that speaker id to doctor or patient using speakerRoleMap when provided, or a conservative dynamic map where the first speaker is defaultHumanRole and the second distinct speaker is the other human role. If no speaker_id is present, the transcript payload remains honest and marks attributionSource=missing-speaker-id; it does not claim acoustic diarization.

Piper TTS is model-file driven. The agent can select a Piper model per room before the LiveKit session is created:

PIPER_MODEL_PATH_ES=/srv/piper/voices/es_MX-claude-high.onnx
PIPER_MODEL_PATH_EN=/srv/piper/voices/en_US-lessac-medium.onnx
PIPER_SPEAKER_ID_EN=
PIPER_SPEAKER_ID_ES=

PIPER_MODEL_PATH is still supported as a legacy language-agnostic fallback, but new deployments should prefer PIPER_MODEL_PATH_EN and PIPER_MODEL_PATH_ES. If the requested language-specific model is missing, the agent logs the actual fallback source, for example piper_voice_source=piper_model_path_es_fallback. The CPU deployment image is expected to include both the Spanish and English Piper models and should set PIPER_MODEL_PATH_EN to the English model path.

A site can override the full session instructions through LiveKit room metadata (agent_prompt) or future site configuration. Code, comments, tests, and public PR descriptions remain in English.

Hackathon Positioning

Recommended short description:

OpenMRS LiveKit is a fully local clinical interpreter and encounter compiler. It runs voice capture, transcription, translation, and structured extraction on clinic hardware, de-identifies text before model inference, and produces clinician-reviewed OpenMRS draft observations without sending PHI to cloud AI services.

Target track: Clinical Track.

Distribution model: fully open source.

License: MIT.

Documentation

• Hackathon dossier
• Submission form fields
• Demo script
• Security model
• Proposal positioning

Status

Working prototype for the OpenMRS AI Hackathon 2026. The repository contains the LiveKit agent foundation and the first OpenMRS-specific safety primitives. It still needs production hardening, a review UI, site-specific concept mapping, and validated local model packaging before clinical use.

License

MIT.