Skip to content

operations.md

Latest commit

 

History

History
385 lines (272 loc) · 9.54 KB

operations.md

File metadata and controls

385 lines (272 loc) · 9.54 KB
Raw

< Architecture | Deep Dive | Quick Start

Operations, Debugging & Metrics

This document covers operational procedures for monitoring, debugging, and maintaining the Self-Hosted RAG system.

DuckDB — Lifecycle Inspection

The chunks.duckdb database (in $DEFAULT_DOC_INGEST_ROOT) is the primary source of truth for all document state.

duckdb $DEFAULT_DOC_INGEST_ROOT/chunks.duckdb

Document Status Overview

SELECT original_filename, status, worker_id, new_at
FROM ingestion_lifecycle
ORDER BY new_at DESC;

Find Stuck Jobs (In progress > 1 hour)

SELECT id, original_filename, status
FROM ingestion_lifecycle
WHERE status NOT LIKE '%SUCCESS%'
  AND status NOT LIKE '%FAILED%'
  AND new_at < (CURRENT_TIMESTAMP - INTERVAL 1 HOUR);

Timing Breakdown (Per Document)

SELECT
    original_filename,
    preprocessing_complete_at - preprocessing_at AS normalization_time,
    ingesting_at - preprocessing_complete_at AS chunking_time,
    consuming_at - ingesting_at AS queue_time,
    finalized_at - consuming_at AS persistence_time,
    finalized_at - new_at AS total_turnaround
FROM ingestion_lifecycle
WHERE status = 'INGEST_SUCCESS';

Inspect Errors

SELECT original_filename, error_log
FROM ingestion_lifecycle
WHERE status = 'INGEST_FAILED'
ORDER BY finalized_at DESC;

Verify Physical File Locations

SELECT status, pdf_path, md_path
FROM ingestion_lifecycle
WHERE original_filename LIKE '%my_document%';

Lock Contention Audit

SELECT slug, status, error
FROM gatekeeper_history
WHERE status = 'FAILURE'
ORDER BY timestamp DESC;

DuckDB — Chunk Distribution

Count Chunks by Type

SELECT type, count(*) AS chunk_count
FROM parquet_chunks
GROUP BY type;

Largest Documents (by Chunk Count)

SELECT source_file, count(*) AS chunks
FROM parquet_chunks
GROUP BY source_file
ORDER BY chunks DESC
LIMIT 10;

Page-Level Distribution

SELECT page, count(*) AS chunks_per_page
FROM parquet_chunks
WHERE source_file = 'my_document.pdf'
GROUP BY page
ORDER BY page ASC;

DuckDB — Staging Inspection

The staged_chunks table acts as the Write-Ahead Log for chunks before Qdrant persistence.

Current Buffer Size

SELECT count(*) AS enqueued_chunks, count(DISTINCT source_file) AS active_files
FROM staged_chunks;

Chunk Size Check (Character-Length Proxy)

SELECT id, length(chunk) AS chars
FROM staged_chunks
ORDER BY chars DESC
LIMIT 10;

Integrity Check — Non-Deterministic IDs

SELECT id, source_file
FROM parquet_chunks
WHERE id NOT LIKE 'DOC_%';

Redis — Queue Inspection

docker exec -it doc-ingest-chat-redis-1 redis-cli
> LLEN chunk_ingest_queue:0
> LLEN chunk_ingest_queue:1
> LLEN ocr_job_queue
> LLEN whisper_job_queue

Note: When USE_TEMPORAL_WHISPER=true, the whisper_job_queue is bypassed — transcription jobs are dispatched to the Temporal task queue instead. Monitor those via the Temporal Web UI (http://localhost:8233) or tctl CLI.

Whisper.cpp — Server Debugging

The whisper.cpp server must be started with --convert to handle non-WAV formats (MP4, MP3, M4A, etc.). Without it, the server returns 400 Bad Request and logs failed to decode audio data from memory buffer.

Verify Server

curl http://<whisper-host>:1145/

Should return an HTML page with the API documentation.

Test with WAV (basic)

curl http://<whisper-host>:1145/inference \
  -F "file=@/path/to/test.wav" \
  -F "temperature=0.0" \
  -F "response_format=json"

Test with MP4 (requires --convert)

curl http://<whisper-host>:1145/inference \
  -F "file=@/path/to/test.mp4" \
  -F "temperature=0.0" \
  -F "temperature_inc=0.2" \
  -F "no_speech_thold=0.6" \
  -F "response_format=json"

Convert MP4 to WAV manually

ffmpeg -i /path/to/test.mp4 -vn -acodec pcm_s16le -ar 16000 -ac 1 /tmp/test.wav

Common Issues

Symptom Cause Fix
400 Bad Request + failed to decode audio data Server missing --convert flag Restart server with --convert
Connection refused Server not running or wrong host/port Check WHISPER_MODEL_ENDPOINTS env var
Transcription empty Audio too quiet or no speech Check no_speech_thold parameter

HAProxy — Load Balancer Monitoring

When multi-endpoint *_ENDPOINTS env vars are set, HAProxy containers handle request distribution across backends.

Stats UI

Service URL
Supervisor LLM http://localhost:8404/stats
Embedding http://localhost:8405/stats
Whisper http://localhost:8406/stats
OCR http://localhost:8407/stats

Check HAProxy Logs

docker logs haproxy_supervisor 2>&1 | grep "POST\|GET" | tail -20
docker logs haproxy_embd 2>&1 | grep "POST\|GET" | tail -20
docker logs haproxy_whisper 2>&1 | grep "POST\|GET" | tail -20
docker logs haproxy_ocr 2>&1 | grep "POST\|GET" | tail -20

Verify Traffic Distribution

Requests should alternate between backends (srv0, srv1, etc.):

docker logs haproxy_supervisor 2>&1 | grep "be_supervisor/" | tail -10

Expected output shows alternating backends:

be_supervisor/srv0 ... "POST /v1/chat/completions HTTP/1.1"
be_supervisor/srv1 ... "POST /v1/chat/completions HTTP/1.1"
be_supervisor/srv0 ... "POST /v1/chat/completions HTTP/1.1"

Check Backend Health

HAProxy health-checks backends every 2s via GET /models (or GET /health). A backend is marked down after 3 consecutive failures and up after 2 successes.

docker exec haproxy_supervisor cat /tmp/haproxy.cfg | grep "server srv"

Common Issues

Symptom Cause Fix
All traffic to one backend Keep-alive connection pinning option httpclose is set — check if client is reusing connections
Backend marked DOWN Health check failing Check if the backend's /models or /health endpoint responds
503 from HAProxy 0 endpoints configured Set *_ENDPOINTS env var or point *_PATH directly to backend

Temporal — Workflow Monitoring (when enabled)

When USE_TEMPORAL_WHISPER=true, the system connects to a remote Temporal server.

Web UI

Access the Temporal Web UI provided by your remote deployment (URL varies by infrastructure).

CLI Monitoring

# List running workflows
tctl --namespace default workflow list

# Describe a specific workflow
tctl --namespace default workflow describe --workflow_id <workflow-id>

# View workflow execution history
tctl --namespace default workflow show --workflow_id <workflow-id>

Common Issues

Symptom Cause Fix
Workflows stuck in PENDING Temporal server unreachable Verify TEMPORAL_HOST and TEMPORAL_PORT are correct and the remote server is reachable
Activities timing out Whisper server down or slow Check WHISPER_MODEL_ENDPOINTS and whisper server health
Transcription not starting USE_TEMPORAL_WHISPER not set Set USE_TEMPORAL_WHISPER=true and restart workers

Qdrant — Vector Inspection

Point Count for a Document

curl -X POST http://<vector-db-host>:6333/collections/vector_base_collection/points/count \
  -H "Content-Type: application/json" \
  -d '{
    "filter": {
      "must": [{"key": "source_file", "match": {"text": "my_document"}}]
    }
  }'

Replace <vector-db-host> with your Qdrant REST API endpoint (default port 6333, or as configured via VECTOR_DB_URL).

Sample Payloads

curl -X POST http://<vector-db-host>:6333/collections/vector_base_collection/points/scroll \
  -H "Content-Type: application/json" \
  -d '{"limit": 3, "with_payload": true, "with_vector": false}'

Qdrant Dashboard

Visit http://<vector-db-host>:6333/dashboard for the built-in web UI.

Metrics (JSONL)

Metrics are recorded in $DEFAULT_DOC_INGEST_ROOT/metrics.jsonl.

Average Normalization Time

jq -r 'select(.event == "file_processing_complete") | .metrics.total_processing_time_ms' \
  $DEFAULT_DOC_INGEST_ROOT/metrics.jsonl | \
  awk '{sum+=$1; count+=1} END {print "Avg: " sum/count " ms"}'

Schema Evolution

The canonical schema is in doc-ingest-chat/sql/schema.sql. Update that file first, then apply changes:

Add a Column

ALTER TABLE ingestion_lifecycle ADD COLUMN language VARCHAR DEFAULT 'en';

Create an Index

CREATE INDEX idx_source_file ON parquet_chunks (source_file);

Wipe All State (Fresh Start)

Warning: This permanently deletes all ingestion state and chunk data. Back up chunks.duckdb before running these commands. Qdrant vectors are not affected by DuckDB deletes and must be cleaned separately.

DELETE FROM ingestion_lifecycle;
DELETE FROM parquet_chunks;
DELETE FROM staged_chunks;
DELETE FROM file_ingestion_jobs;
DELETE FROM gatekeeper_history;

War Room Scenarios

"Data not found" but file was ingested

SELECT status, error_log, pdf_path, md_path
FROM ingestion_lifecycle
WHERE original_filename = 'missing_file.pdf';

DuckDB and Qdrant out of sync

-- Find chunks in DuckDB without deterministic IDs
SELECT id, source_file
FROM parquet_chunks
WHERE id NOT LIKE 'DOC_%';

Ingestion stalled

Check for lock contention using the query in Lock Contention Audit above. Also check Redis queue lengths — if LLEN chunk_ingest_queue:N is growing without bound, the Consumer may have crashed or be blocked.