Steve Chat Web (Prototype)

Mobile-first local chat UI prototype inspired by Allan's sketch (hamburger menu, model picker, simple message flow).

Current scope

• Step 1: repo scaffold ✅
• Step 2: mobile shell/UI ✅
• Step 3: tap-through chat flow ✅
  • chat drawer + searchable chat history
  • model picker sheet
  • simplified settings sheet (single list with backend dropdown, model profile dropdown, chat defaults)
  • message composer + mock Steve replies + mock mic button
  • assistant metadata includes live tokens/s + power telemetry
  • thinking-capable models render inline <think>...</think> output in a live toggleable Thinking bubble
  • local-first speech-to-text settings for Whisper small/medium/large models
  • bottom session token counters (total / prompt / completion)
  • optional live wire-up to local OpenAI-compatible endpoint (/v1/models, /v1/chat/completions)
  • foldable/wide layout: persistent left chat drawer + right chat pane
  • collapsible wide-mode sidebar rail with deterministic SHA-256 identicon artwork per chat

Run locally

bash serve.sh 8104

Open:

• http://127.0.0.1:8104

Live mode

From the Settings button (⚙):

1. Set base URL (default http://127.0.0.1:18084)
2. Pick backend from dropdown (Regular, QVAC fabric, or PrismML Vulkan)
3. Pick local model profile (Gemma, Ternary Bonsai, or LFM2.5) and tap Apply model
4. Tap Detect to load local models
5. Tune chat sampling/perf settings in Chat defaults:
   • temperature, top-k, top-p, min-p, typical-p, repeat penalty, max tokens
   • optional custom JSON (advanced) merged into /v1/chat/completions
6. Choose mode:
   • UI Demo (mock Steve replies)
   • Local Runtime (real endpoint calls)

Default profile now prefers QVAC Vulkan (GPU) + Local Runtime for on-device acceleration testing. You can switch back to UI Demo any time for pure UX review.

Local llama.cpp quick start

This repo includes a helper script to run either backend:

• regular llama-server prebuilt on 127.0.0.1:18080
• regular local CPU build on 127.0.0.1:18082
• regular local Vulkan build on 127.0.0.1:18083
• qvac CPU build on 127.0.0.1:18081
• qvac Vulkan build on 127.0.0.1:18084
• PrismML llama.cpp Vulkan build on 127.0.0.1:18092

./scripts/llama_cpp_local.sh list-models
./scripts/llama_cpp_local.sh start --backend regular --mode gpu --index 1
./scripts/llama_cpp_local.sh start --backend regular --mode cpu --index 1

# launch additional regular endpoints from custom binaries
LLAMA_CPP_BIN=/tmp/llama-b8419/build-openclaw-cpu/bin/llama-server LLAMA_CPP_PORT=18082 \
  ./scripts/llama_cpp_local.sh start --backend regular --mode cpu --index 1
LLAMA_CPP_BIN=/tmp/llama-b8419/build-openclaw-vulkan/bin/llama-server LLAMA_CPP_PORT=18083 \
  ./scripts/llama_cpp_local.sh start --backend regular --mode gpu --index 1

Switch model (example E2B → E4B):

./scripts/llama_cpp_local.sh restart --backend regular --mode gpu --index 2

Start qvac backend (if qvac binary is installed):

./scripts/llama_cpp_local.sh start --backend qvac --mode gpu --index 1

Start all runtime variants at once (high RAM use; may be unstable on constrained devices):

./scripts/start_runtime_matrix.sh

Recommended for phone testing: run one target at a time (stops others first):

./scripts/switch_runtime_target.sh reg-vulkan
./scripts/switch_runtime_target.sh qvac-vulkan
MODEL_INDEX=9 ./scripts/switch_runtime_target.sh prism-vulkan

Then in Steve Chat Settings:

1. choose backend (Regular, QVAC, or PrismML)
2. choose model profile (Gemma, Ternary Bonsai, or LFM2.5) and tap Apply model
3. tap Connect local … to set endpoint + detect models.

Gemma 4 E2B IT QAT is available as profile g4e2bQat / model index 14:

• g4e2bQat: /storage/emulated/0/OpenClawHub/models/gemma-4-E2B-it-qat-UD-Q4_K_XL.gguf

The Ternary Bonsai profiles are mapped to the PrismML Vulkan runtime:

• bonsai17b / model index 9: /root/.openclaw/workspace/models/prismml/Ternary-Bonsai-1.7B-Q2_0.gguf
• bonsai4b / model index 10: /root/.openclaw/workspace/models/prismml/Ternary-Bonsai-4B-Q2_0.gguf
• bonsai8b / model index 11: /root/.openclaw/workspace/models/prismml/Ternary-Bonsai-8B-Q2_0.gguf

The LFM2.5 1.2B profiles are also mapped to the PrismML Vulkan runtime:

• lfm25Instruct12b / model index 12: /data/data/com.termux/files/home/models/LFM2.5-1.2B-Instruct-Q4_K_M.gguf
• lfm25Thinking12b / model index 13: /data/data/com.termux/files/home/models/LFM2.5-1.2B-Thinking-Q4_K_M.gguf
• lfm25MoE8bA1b / model index 15: /storage/emulated/0/OpenClawHub/models/LFM2.5-8B-A1B-Q4_K_M.gguf
  • Download: https://huggingface.co/LiquidAI/LFM2.5-8B-A1B-GGUF/resolve/main/LFM2.5-8B-A1B-Q4_K_M.gguf

The LFM2.5 Thinking profile is parsed as a thinking-capable model when the runtime returns OpenAI-style reasoning_content / thinking fields or inline <think>...</think> / <|START_THINKING|>...<|END_THINKING|> text.

Local speech-to-text

serve.sh starts the local STT sidecar on 127.0.0.1:18777 when the STT venv exists. In Settings, use Speech to text to:

1. choose Whisper small.en (~465 MB), medium.en (~1.43 GB), or large-v3 (~3.1 GB),
2. open the selected Whisper browser download link,
3. use Browse to select the download/model folder when the browser or Android bridge exposes it,
4. persist the STT endpoint and model directory,
5. record from the composer and transcribe the completed recording locally.

The browser app records with MediaRecorder and posts the final audio blob to /stt/transcribe. Browser SpeechRecognition is no longer the primary transcription path.

Direct STT downloads use whisper.cpp .bin files from ggerganov/whisper.cpp; the current Python sidecar still transcribes with faster-whisper, so native/Android wiring may need to map the selected browser folder into a whisper.cpp-capable transcription backend. The sidecar installer endpoint remains available for faster-whisper cache installs.

Build helper for upstream llama.cpp arm64 CPU/Vulkan artifacts: scripts/phase2b_build_llama_org_arm64.sh

Runtime packaging note: keep libmtmd.so* beside llama-server (plus libllama.so* / libggml*.so*) or launch with LD_LIBRARY_PATH including that folder.

Note: bundled server artifacts must include shared libs beside the binary (including libmtmd.so*). llama_cpp_local.sh now prepends the binary directory to LD_LIBRARY_PATH automatically.

Detailed guide: docs/PHASE2-LLAMA-CPP-SETUP.md

UI naming map: docs/UI-ELEMENT-NAMING.md

Next iteration ideas

• background endpoint health monitor + reconnect UX
• chat export/import
• model profile presets (ctx, temp, max tokens)
• avatar/theme customization ("Steve" personality presets)