SpeechTools is a production-grade, enterprise multi-tenant SaaS platform and autonomous AI agent loop engineered for high-throughput Indic speech annotation, automated boundary correction, intelligent segmentation, transcription, quality control (QC), and self-healing remediation.
The system addresses the critical bottlenecks in large-scale speech dataset annotation across 7 Indian languages:
🇮🇳 Hindi (hi-IN), 🇮🇳 Marathi (mr-IN), 🇮🇳 Bengali (bn-IN), 🇮🇳 Telugu (te-IN), 🇮🇳 Kannada (kn-IN), 🇮🇳 Tamil (ta-IN), and 🇮🇳 English (en-IN).
It operates under a dual-architecture paradigm:
- Cloud Multi-Tenant SaaS Platform: Asynchronous FastAPI backend, PostgreSQL 16 with Row-Level Security (RLS) forced on all tenant tables, Redis 7 Lua-scripted distributed semaphore, Arq task queue with mid-task crash recovery, Argon2id security pool (<= 256 MiB RAM), and Caddy TLS reverse proxy.
- Autonomous Multi-Agent Loop (LECS): Supervisory Loop Engineering Control System, Playwright Chromium headless automation, SQLite adaptive rule memory (
memory/rules.db) with 110 canonical rules, hardware-fingerprinted zero-knowledge credential vault (security/credential_vault.py), and automated End-of-Day (EOD) reporting.
- 1. Dual-Architecture Overview
- 2. Chronological Engineering Journey (From Start to Finish)
- 3. Core Architecture & Security Models
- 4. Empirical Benchmarks & Measured Experiments (E1 to E9, A4.4)
- 5. Test Traceability & Taxonomy (100% Passing)
- 6. Production Execution & Live Remediation Proofs
- 7. Commercial Monetization & Licensing Engine
- 8. What Was Done vs What Remains (Roadmap / Next Gates)
- 9. Setup, Installation & Verification Guide
- 10. Repository Directory Map
SpeechTools solves two operational modes simultaneously:
┌──────────────────────────────────────────────┐
│ SPEECHTOOLS CORE PLATFORM │
└──────────────────────┬───────────────────────┘
│
┌────────────────────────────────┴───────────────────────────────┐
▼ ▼
┌────────────────────────────────────────┐ ┌────────────────────────────────────────┐
│ ENTERPRISE MULTI-TENANT CLOUD SAAS │ │ AUTONOMOUS WORKER AGENT & DESKTOP │
├────────────────────────────────────────┤ ├────────────────────────────────────────┤
│ • Caddy TLS Reverse Proxy (Trusted XFF)│ │ • 1-Click Desktop Runner (START.bat) │
│ • FastAPI (Async SQLAlchemy + asyncpg) │ │ • Local Control Plane API (Port 3001) │
│ • PostgreSQL 16 RLS (Default Deny) │ │ • Playwright Browser Automation Loop │
│ • Redis 7 Distributed Semaphore (Cap=8)│ │ • SQLite Persistent Rule Memory (110) │
│ • Arq Async Worker Queue (Auto Requeue)│ │ • Hardware Zero-Knowledge Vault (AES) │
│ • Argon2id Worker Pool (<= 256 MiB RSS)│ │ • Automated EOD WhatsApp/Slack Ledger │
│ • React + TypeScript + Vite Frontend │ │ • 27-Task Free Trial & License Guard │
└────────────────────────────────────────┘ └────────────────────────────────────────┘
Every phase of SpeechTools was developed, empirically measured, and committed through strict gate-based verification:
- Initialized multi-container orchestration (
docker-compose.yml,docker-compose.cloud.yml). - Configured Caddy reverse proxy with automatic HTTPS and proxy header forwarding.
- Structured least-privilege Docker containers with non-root security contexts (
UID 1000).
- Established strict PostgreSQL role separation:
speechtools_admin: Migration runner withBYPASSRLS.speechtools_app: Runtime application role withNOBYPASSRLSand forced row-level filtering.
- Implemented Kubernetes-style
/healthzand/readyzprobes verifying deep connectivity to PostgreSQL and Redis.
- Built state-of-the-art authentication with Argon2id password hashing.
- Implemented secure HTTP-only,
SameSite=Laxcookie emission with anti-CSRF double-submit token headers. - Engineered refresh token rotation with Theft Detection & Family Revocation (replaying a rotated token terminates the entire session).
- Calibrated SlowAPI rate limiting (IP-based) and exponential backoff account lockouts (5 bad attempts -> 15 min lock).
- Verified with empirical benchmarks: Experiments E1, E2, E3, E4, E5.
- Migrated multi-tenant schema with PostgreSQL 16 Row-Level Security (RLS) forced across all tenant tables:
vault_credentials,tasks,sessions, andaudit_logs. - Implemented session-level context injection via
set_config('app.current_tenant_id', :tenant_id, true). - Designed two-tier envelope encryption: Random 256-bit DEK per secret, encrypted with host KEK via AES-256-GCM.
- Verified authenticated ciphertext tamper detection and large payload handling (Playwright session state up to 512KB).
- Verified with empirical benchmarks: Experiments E6 & E7.
- Gate 0 & 1: Formulated the non-negotiable Testing Constitution (
TRUTH > SAFETY > VERIFICATION > RECOVERY > THROUGHPUT) and universal verification harness (scripts/verify_all.py). - Gate 2: Established test taxonomy markers (
@pytest.mark.unit,@pytest.mark.integration,@pytest.mark.e2e) and mapped full requirements indocs/TEST_MATRIX.md. - Gate 3: Enforced strict test database isolation on
speechtools_testwith non-bypassable RLS break-it proofs. - Gate 4: Implemented Redis distributed semaphore (
app/services/distributed_semaphore.py) enforcing global atomic limits (MAX_BROWSERS=8, max 4 concurrent tasks per tenant). - Gate 5: Re-benchmarked Argon2id in a bounded
ThreadPoolExecutor(max_workers=4)ensuring peak memory overhead strictly <= 256 MiB, constant-timecompare_digestcomparisons, and zero user enumeration. Verified via Experiment A4.4.
- Codified 110 verified rules into
memory/rules.dband CANONICAL_GUIDELINES.md. - Deployed autonomous
agent_orchestrator.pyandportal_agent.pyon the live annotation portal:- Processed 191 tasks: 180 submitted, 10 reported (script mismatch), 1 failed.
- Successfully remediated 9 historically rejected QC tasks with 0 errors on AWS validation gate (REMEDIATED_TASKS_PROOF.md).
- Built 1-Click desktop distribution (
START.bat,local_api.py,ui/) with license gating and direct UPI monetization.
All sensitive tables (vault_credentials, tasks, sessions, audit_logs) enforce PostgreSQL Row-Level Security (ALTER TABLE ... FORCE ROW LEVEL SECURITY):
-- RLS Policy Example on vault_credentials
CREATE POLICY tenant_isolation_policy ON vault_credentials
FOR ALL
TO speechtools_app
USING (tenant_id = current_setting('app.current_tenant_id', true)::uuid)
WITH CHECK (tenant_id = current_setting('app.current_tenant_id', true)::uuid);- Fail-Closed Default Deny: If a query executes without setting
app.current_tenant_id, PostgreSQL returns 0 rows, preventing cross-tenant leakage even if application code omits aWHEREclause. - Role Boundary:
speechtools_appdoes NOT possessSUPERUSERorBYPASSRLS. Onlyspeechtools_admincan bypass RLS during migrations.
To prevent Playwright browser explosions across multiple horizontally scaled worker containers:
- Implemented Lua-scripted atomic slot lease and heartbeat in Redis:
- Global browser cap:
MAX_BROWSERS = 8. - Per-tenant queue limit:
MAX_TASKS_PER_TENANT = 2(excess tasks receive HTTP 429). - Slot leases expire after 300s if worker crashes, preventing orphaned semaphore deadlocks.
- Global browser cap:
-- Atomic slot acquisition with expiration check
local current = redis.call('ZCARD', KEYS[1])
if current < tonumber(ARGV[1]) then
redis.call('ZADD', KEYS[1], ARGV[2], ARGV[3])
return 1
else
return 0
endArgon2id parameters calibrated to balance cracking resistance against denial of service:
-
time_cost: 4 iterations -
memory_cost: 65,536 KiB (64 MiB) -
parallelism: 4 lanes -
Memory Ceiling: Wrapped in a bounded
ThreadPoolExecutor(max_workers=4)gated byasyncio.Semaphore(4). Even under 500 concurrent login attempts, maximum RSS overhead for password hashing is capped at:$$\text{Peak Memory} = 4 \times 64,\text{MiB} = 256,\text{MiB}$$
Local desktop and cloud credentials use AES-256-GCM envelope encryption:
- Hardware-fingerprinted machine key (SHA-256 derived from motherboard UUID + CPU ID via PBKDF2 with 600,000 iterations).
- Generates a random 256-bit Data Encryption Key (DEK) per secret.
- Plaintext is encrypted with DEK (AES-256-GCM, 12-byte random IV, 16-byte authentication tag).
- DEK is encrypted with Key Encryption Key (KEK).
- Authenticated tag verification detects any bit manipulation or tampering immediately.
The agent features persistent SQLite memory with an adaptive feedback loop:
- Base Rules: 146 canonical rules parsed directly from CANONICAL_GUIDELINES.md.
- Learned Rules: Automatically extracted from QC rejection comments.
- Pattern Promotion: If 3 or more users encounter the same mistake pattern, the rule is promoted to
GLOBAL_CRITICALstatus. - Rules are NEVER forgotten across restarts.
All values below were measured from real code execution on the host system:
| Experiment | Focus Area | Metric Measured | Target Threshold | Measured Result | Verdict |
|---|---|---|---|---|---|
| E1 | Argon2id Calibration | Hashing Latency (ms) | 200–350 ms | 239.23 ms (StdDev: 24.49 ms) | PASS |
| E2 | Rate Limiting & Lockout | 100 IP flood & 20 IP brute | 429 on flood; Lock at attempt 5 | 90/100 HTTP 429; Lockout at attempt 5 | PASS |
| E3 | High Concurrency Load | 500 concurrent signups/logins | Zero 500 errors | 499/500 HTTP 200/201 (0 server errors) | PASS |
| E4 | Token Replay Attack | Replay rotated refresh token | Immediate session revocation | HTTP 401; entire token family revoked | PASS |
| E5 | Postgres Chaos Injection | Kill DB container mid-request | 503 during outage; fast healing | Clean 503; self-healed in < 2.94s | PASS |
| E6 | AES-256-GCM Envelope | Crypto Roundtrip Latency (ms) | p95 < 5.0 ms | Mean: 0.123 ms | p95: 0.279 ms | PASS |
| E7 | PostgreSQL RLS Overhead | Direct vs RLS Query (ms) | Overhead < 3.0 ms | Baseline 1.71ms vs RLS 2.70ms (+0.99ms) | PASS |
| E8 | Playwright Process RSS | Memory per Browser Context | < 2.5 GB at max load | 1 context: 529MB | 8 contexts: 1,177MB (+91.5MB/ctx) | PASS |
| E9 | Queue Acquisition Latency | Monotonic wait at cap (ms) | Backlog begins immediately | Under cap: 0.01ms | Past cap (15 tasks): 46.90ms mean | PASS |
| A4.4 | Multi-Worker Scaling | 40-task burst across 3 workers | Peak concurrency <= 8 | Peak: exactly 8 active browsers; 40/40 completed | PASS |
Full experimental methodology and raw outputs: docs/EXPERIMENTS.md
Organized strictly by taxonomy markers:
@pytest.mark.unit: 14 tests (in-memory settings validator, trace sanitizers, payload caps).@pytest.mark.integration: 62 tests (real PostgreSQL 16 with RLS, real Redis 7, Alembic migrations, token rotation).@pytest.mark.e2e: 1 test (real Playwright Chromium browser launch, cross-tenant context & cookie isolation).- Result: 77/77 PASSED (100%).
5 consecutive end-to-end verification runs testing:
- Machine Fingerprint stability (SHA-256 64-char hex).
- AES-256-GCM encrypt/decrypt roundtrip, tampering, and 10KB payloads.
- Hardware Credential Vault CRUD and wrong-PIN rejection.
- Adaptive Rule Memory parsing 146 base rules, learning 5 feedback rules, and persisting across re-opens.
- Honesty Protocol detecting uncertainty and missing guidelines.
- License Guard offline handling and trial limit tamper detection.
- License Server DB user/license/referral verification.
- Complete integration pipeline simulation.
- Result: 87/87 PASSED across all 5 verification runs (100%).
- Verified all 16 QC gates across Hindi, Marathi, Bengali, English, Tamil, Kannada, and Telugu.
- Negative test verified: Script mismatch (e.g. Devanagari in Bengali task) immediately caught and routed to 'Not Target Language' reporting.
- Result: 100% PASSED.
- Verified all 9 endpoints (
/,/api/dashboard,/api/memory/rules,/api/vault/info,/api/task-types,/api/analytics,/api/license/status,/api/memory/feedback,/api/settings). - Result: 100% PASSED.
The autonomous agent operated on the live speech annotation portal (https://sbs.aidicrowd.com) with the following verified record:
Total Processed : 191 Tasks
✅ Submitted : 180 Tasks (100% AWS QA Gate Pass Rate)
⚠️ Reported/Skip: 10 Tasks (Non-target language / No auto-transcript)
❌ Failed : 1 Task
Every submitted task satisfied all mandatory domain rules:
-
Segment Duration: Strictly
$< 30.0,\text{s}$ . -
Boundary Gap: Strictly
$\ge 1,\text{ms}$ between consecutive segments (no shared timestamps). -
Zero Hindi Danda: Strictly eliminated all
।and|, replacing with sentence-ending punctuation (.,?,!). -
Foreign Spoken Words: Enclosed in
<UNKNOWN/>tags. -
Personal Names: Tagged with
<PName> ... </PName>with zero artificial letter-splitting.
Detailed task-by-task execution log: TASK_EXECUTION_REPORT.md
9 tasks previously rejected by the human QC team were fed to the autonomous remediation engine. The agent extracted the feedback, updated memory/rules.db, adjusted the timestamps, corrected tags, validated via QA sub-agent, and submitted with photographic proof:
| # | Task ID | File Name | QC Rejection Reason | Remediation Applied | Status |
|---|---|---|---|---|---|
| 1 | 1787207895405 |
..._seg3.m4a |
English words transliterated; Hindi danda used; boundary overlap | Applied <UNKNOWN/>, replaced । with ., enforced 2ms gaps |
✅ SUBMITTED |
| 2 | 1787207895407 |
..._seg4.m4a |
English words transliterated into Devanagari; punctuation | Tagged English lexicon with <UNKNOWN/>, verified Devanagari script |
✅ SUBMITTED |
| 3 | 1787207895409 |
..._seg5.m4a |
Guidelines violation on profit/trend; sentence endings | Normalized ITN, added proper end punctuation | ✅ SUBMITTED |
| 4 | 1787207895411 |
..._seg6.m4a |
Infrastructure, Capex, Service Sector transliterated | Tagged spoken English terms with <UNKNOWN/> |
✅ SUBMITTED |
| 5 | 1787207895413 |
..._seg7.m4a |
BlackRock transliteration; offensive term handling | Remediated entity spelling, applied acoustic review | ✅ SUBMITTED |
| 6 | 1787207895415 |
..._seg8.m4a |
Ratio, acceleration transliterated; missing tags | Tagged spoken technical terms with <UNKNOWN/> |
✅ SUBMITTED |
| 7 | 1787207895417 |
..._whole.m4a |
Recipe share, unique, my channel transliterated | Applied <UNKNOWN/> to all conversational loan words |
✅ SUBMITTED |
| 8 | 1787207895419 |
..._whole.m4a |
Missing <PName> tags on person names; spelling |
Enclosed all names in <PName> ... </PName> |
✅ SUBMITTED |
| 9 | 1787207895421 |
..._whole.m4a |
Missing name tags on historical figures / channels | Identified named entities, tagged <PName> |
✅ SUBMITTED |
Complete remediation audit and photographic diffs: REMEDIATED_TASKS_PROOF.md
The system maintains an automated ledger of all reported tasks in reported_tasks_eod.json. At the end of each shift, running:
python agent_orchestrator.py --eod-reportoutputs the formatted EOD submission block for WhatsApp and Slack groups:
============================================================
📋 EOD REPORT FOR TC TASK ISSUES
============================================================
1. Not Target Language:
- 029313ad-2540-5f90-bb49-fb772a73c0c6_seg0.m4a
- 02ac0ca4-cf4f-58fb-9eb5-748532e4eb24_seg1.m4a
2. No Auto Transcription:
- 1540e5d4-5351-5000-ab96-6f27d1e4610f_seg4.m4a
3. Reco Not in Target Language:
(Skipped per protocol)
============================================================
SpeechTools includes a built-in commercial distribution model designed for local execution with zero cloud hosting bills:
- 1-Click Portable Distribution: Entire app bundled into a shareable distribution with
START.bat(Windows) andSTART_MAC.command(macOS). - Free Batch (27 Tasks): Every new machine gets 27 free tasks out-of-the-box.
- Paywall & UPI Payment: On Task #28, the app pauses and displays a direct UPI QR payment modal:
- Payee: Arsh Malik (HDFC Bank) — 0% payment gateway commission.
- Pro Plan: ₹2,000/mo (or ₹1,500/mo with referral code
SPECIAL500).
- Offline / Local Verification: License validation works with HMAC-SHA256 signature verification tied to the hardware fingerprint. Tamper detection automatically enforces trial limits if local files are manipulated.
- Gates 0–5: Foundation, Playwright worker container, test taxonomy, test DB isolation, distributed Redis semaphore, Argon2 bounded pool, and anti-enumeration.
- Multi-Tenant SaaS: PostgreSQL 16 RLS, Alembic migrations, Caddy reverse proxy, JWT token family rotation, SlowAPI rate limiting.
- Local Control Plane & UI:
local_api.py, Vite UI dashboard, credentials vault, EOD reporting, learning center. - 110-Rule Adaptive Memory: SQLite persistent engine (
memory/rules.db) with zero-forgetting guarantees. - Live Portal Execution: 191 tasks executed, 180 submitted, 100% QA pass rate.
- QC Remediation: 9 rejected tasks remediated and verified.
- Comprehensive Benchmarks: Experiments E1 through E9 and A4.4 fully documented with raw data.
- 100% Tests Passing: 77 Pytest tests + 87 System tests + 7 Multi-language tests.
-
Gate 6: Coverage enhancement to
$\ge 85%$ overall and$\ge 95%$ on critical auth/vault modules, plus Mutation Testing (mutmut$\ge 80%$ mutants killed). -
Gate 7: Phase 3b.1 Mock Portal (
mockportalservice, side-effect log, fault injection API). -
Gate 8: Phase 3b.2 Formal Loop Engine (
PLAN -> ACT -> OBSERVE -> VERIFY -> DECIDE). - Gate 9: Phase 3b.4 Chaos & Integration tests (lying portal break-it proof, kill-point chaos).
- Gate 10: Phase 3b.5 Experiments E8b, E9b, and E10 (browser lifecycle ADR, queue latency, 300 faults).
-
Gate 11: Universal verification run (
python scripts/verify_all.py), single consolidated summary, commit & STOP.
# Windows: Double-click START.bat or run:
START.bat
# macOS: Double-click START_MAC.command or run:
bash start_mac.sh
# Open browser to access local UI:
http://localhost:3001# 1. Generate secrets (or copy .env.example)
cp .env.example .env
# 2. Build and start containers
docker compose up -d
# 3. Check health of all services
docker compose ps
# 4. Run database migrations
docker exec -it speechtools_backend alembic upgrade head# Run all 77 backend pytest tests:
pytest -v
# Run backend tests categorized by taxonomy:
pytest -m unit
pytest -m integration
pytest -m e2e
# Run commercial system 5x verification suite:
python test_full_system.py
# Run local API endpoint tests:
python test_local_api.py
# Run multi-language QC validation tests:
python test_qc_rules.py
python test_mock_pipeline.py# Run autonomous portal pipeline:
python agent_orchestrator.py
# Generate End-of-Day report for WhatsApp/Slack:
python agent_orchestrator.py --eod-reportspeech-tools/
├── backend/ # FastAPI Enterprise Multi-Tenant Backend
│ ├── alembic/ # Database migration versions
│ ├── app/ # Application core, models, schemas, services, API routes
│ │ ├── core/ # Security, config, settings validator, database session
│ │ ├── models/ # SQLAlchemy 2.0 multi-tenant models with RLS
│ │ ├── services/ # Auth, Vault, Distributed Semaphore, Task Queue
│ │ └── api/ # v1 API routers (auth, vault, tasks, health)
│ └── tests/ # 77 Pytest test files (unit, integration, e2e)
├── frontend/ # Enterprise Cloud SaaS Web App (React + TS + Vite)
├── ui/ # Local Desktop Control Plane UI (Port 3001)
├── memory/ # Adaptive Learning Memory Engine
│ └── rule_memory.py # SQLite rules engine (110 canonical guidelines)
├── security/ # Zero-Knowledge Credential Vault & License Guard
│ ├── credential_vault.py # AES-256-GCM hardware-fingerprinted vault
│ └── license_guard.py # Trial tracking, paywall gating, HMAC license checks
├── license-server/ # Central License & UPI Verification Server
│ └── main.py # License issuance, payment webhook, referral system
├── docs/ # Engineering Evidence & Test Registers
│ ├── EXPERIMENTS.md # Raw logs for Experiments E1–E9 & A4.4
│ ├── TEST_MATRIX.md # Traceability matrix across all requirements
│ ├── MICRO_DETAILS.md # Edge-case register (MD-01 through MD-22)
│ └── evidence/ # Gate execution logs (gate0 through gate10)
├── caddy/ # Caddy reverse proxy config with HTTPS & trusted XFF
├── scripts/ # Verification, migration, and benchmarking scripts
│ └── verify_all.py # Cross-platform single-command verification harness
├── experiments/ # Standalone experimental benchmarks (CSV/JSON output)
├── CANONICAL_GUIDELINES.md # Official Speech Annotation Guidelines (Source of Truth)
├── TASK_EXECUTION_REPORT.md # 191 tasks live execution ledger & QA report
├── REMEDIATED_TASKS_PROOF.md # Proof report for 9 remediated QC rejected tasks
├── USER_INSTRUCTION_GUIDE.md # Hindi/English step-by-step user manual
├── agent_orchestrator.py # Main autonomous multi-agent loop orchestrator
├── portal_agent.py # Playwright automation engine for annotation portal
├── local_api.py # Local FastAPI control plane server
├── START.bat # 1-Click Windows startup launcher
├── START_MAC.command # 1-Click macOS startup launcher
└── docker-compose.yml # Multi-container orchestration definition
Copyright © 2026 Arsh Malik. All rights reserved.
Commercial proprietary software. Unauthorized copying, distribution, modification, or commercial exploitation is strictly prohibited without explicit written license from the copyright owner.