Feat/workspaces

.env.example

@@ -27,11 +27,13 @@ FRONTEND_URL=
 PUBLIC_POSTHOG_PROJECT_TOKEN=
 PUBLIC_POSTHOG_HOST=https://us.i.posthog.com
 
-# ── Connection encryption (advanced) ──
-# Encryption key for storing database connection strings at rest.
-# If unset, a key is auto-generated and stored in ~/.bolodb/.secret.
-# RECENT_CONNECTIONS_MASTER_KEY=   # Master key for key rotation (encrypts the key file)
-# RECENT_CONNECTIONS_KEY=          # Direct encryption key (alternative to master key)
+# ── Connection encryption (required for storing DB URLs at rest) ──
+# Encrypts saved connection URLs. REQUIRED — the app will not start without it,
+# and saved connections cannot be stored or reopened until it is set. It is read
+# only from the environment (never written to disk), so keep it stable across
+# deploys: change it and previously saved connections must be re-added.
+# Generate with: python -c "import secrets; print(secrets.token_urlsafe(48))"
+RECENT_CONNECTIONS_KEY=
 
 # ── Server (advanced) ──
 # Set to "true" when behind HTTPS termination (nginx with SSL)
@@ -42,3 +44,9 @@ SKIP_DB_LOCK=false
 # CORS_ORIGINS=
 # Set automatically in Docker; controls SQLite path selection
 # RUNNING_IN_DOCKER=true
+
+# ── Maintenance & Storage ──
+ACTIVITY_LOG_RETENTION_DAYS=30
+ACTIVITY_CLEANUP_ENABLED=true
+ACTIVITY_CLEANUP_INTERVAL_HOURS=24
+BOLODB_DATA_DIR=

.github/workflows/ci.yml

@@ -27,7 +27,10 @@ jobs:
     - name: Install dependencies
       run: |
         python -m pip install --upgrade pip
-        pip install ruff
+        # Pin to match .pre-commit-config.yaml so CI and local hooks lint
+        # identically; an unpinned install pulls whatever ruff released last,
+        # whose changing defaults have broken CI before.
+        pip install ruff==0.15.22
         pip install -r backend/requirements.txt
 
     - name: Lint with Ruff

.gitignore

@@ -18,3 +18,7 @@ credentials.json
 *.key
 .credentials
 docker-compose.override.yml
+venv/
+
+.opencode/
+.agents/

CONTRIBUTING.md

@@ -12,7 +12,7 @@ BoloDB is built using a modern containerized stack designed for ease of local de
 - **Backend:** FastAPI (Python 3.10+) utilizing SQLAlchemy for database connectivity, and `sqlglot` for secure AST-based read-only validation.
 - **Frontend:** A SvelteKit application utilizing TypeScript and Tailwind CSS for a polished, responsive user interface.
 - **Database:** PostgreSQL container storing user accounts, query history, conversations, and recent connections.
-- **Knowledge Base:** Per-user SQLite database storing verified queries, glossary, and trust level.
+- **Knowledge Base:** PostgreSQL persistence storing verified queries, glossary, and trust level.
 - **Reverse Proxy:** Nginx configured to route traffic seamlessly between the frontend and backend services while handling proxy configurations.
 - **LLM Connectors:** Support for OpenRouter API.
 

DOCKERFILE.render

@@ -45,9 +45,14 @@ COPY --from=frontend-builder /app/build /usr/share/nginx/html
 COPY nginx.conf.render /etc/nginx/conf.d/default.conf
 
 RUN addgroup --system bolodb && adduser --system --home /home/bolodb --ingroup bolodb bolodb \
-    && mkdir -p /home/bolodb/.bolodb \
-    && chown -R bolodb:bolodb /home/bolodb/.bolodb /app
+    && mkdir -p /app/data \
+    && chmod +x /app/backend/docker-entrypoint.sh \
+    && chown -R bolodb:bolodb /app
 
 EXPOSE 80
 
+# Same as the compose image: migrate to head before serving, so pointing
+# DATABASE_URL at a new database needs no manual step.
+ENTRYPOINT ["/app/backend/docker-entrypoint.sh"]
+
 CMD sh -c "nginx -g 'daemon off;' & exec python -m uvicorn backend.app.server:create_app --factory --host 0.0.0.0 --port 4321 --log-level info"

LICENSE

@@ -1,6 +1,6 @@
 MIT License
 
-Copyright (c) 2026 HAAHIT
+Copyright (c) 2026 BoloDB
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

README.md

@@ -4,102 +4,98 @@
 
 **Ask your data. Trust the answer.**
 
-A text-to-SQL product for non-technical users. Connect any SQL database, ask
-questions in plain English, get answers with a plain-English explanation and a
-confidence level. Every answer you confirm teaches it your database — accuracy
-improves with use.
+A multi-tenant text-to-SQL web application for non-technical users and teams. Connect your database, ask questions in plain English, and get instant answers with plain-English restatements, ECharts visualizations, and confidence levels. Save queries to interactive dashboards and manage multi-user workspaces with role-based access control (RBAC).
 
-**📚 Full documentation lives in [`docs/`](docs/README.md)** — written for
-non-technical readers, with code pointers for developers at every step.
+**📚 Full documentation lives in [`docs/`](docs/README.md)** — written for non-technical readers, with code pointers for developers at every step.
+
+---
 
 ## Quick Start (Docker)
 
-The easiest and recommended way to run BoloDB is using Docker. This ensures all services (FastAPI Backend, SvelteKit Frontend, Nginx, and MongoDB) run seamlessly.
+The easiest and recommended way to run BoloDB is using Docker Compose (FastAPI backend + SvelteKit frontend + Nginx + PostgreSQL).
+
+1. Install [Docker Desktop](https://www.docker.com/products/docker-desktop/) or Docker Engine.
+2. Clone the repository and navigate into the root directory.
+3. Copy `.env.example` to `.env` and fill in required secrets (`OPENROUTER_API_KEY`, `JWT_SECRET`, `DATABASE_URL`, `RECENT_CONNECTIONS_KEY`, `SUPABASE_URL`, `SUPABASE_ANON_KEY`, `SUPABASE_JWT_SECRET`).
+4. Start the application stack:
 
-1. Install [Docker Desktop](https://www.docker.com/products/docker-desktop/) (Windows/Mac) or Docker Engine (Linux).
-2. Open a terminal in the project directory.
-3. Start the application:
+   **Production / Deployment**:
    ```bash
-   docker compose up -d
+   docker compose up --build -d
    ```
-4. Open [http://localhost:5173](http://localhost:5173) in your browser.
-5. Set the `OPENROUTER_API_KEY` environment variable in your docker-compose or `.env` file, connect a database (or click "Try with sample data"), and start asking!
+   Open [http://localhost:8080](http://localhost:8080).
 
-## The AI engine
+   **Local Development (Vite HMR)**:
+   ```bash
+   docker compose -f docker-compose.yml -f docker-compose.dev.yml up --build
+   ```
+   Open [http://localhost:8080](http://localhost:8080) (Nginx proxy) or [http://localhost:5174](http://localhost:5174) (Vite frontend).
 
-BoloDB uses **OpenRouter** (specifically the `deepseek/deepseek-v4-flash` model) for all AI operations. You only need one thing:
-an OpenRouter API key.
+---
 
-1. Go to https://openrouter.ai/keys.
-2. Click **Create Key** and copy it.
-3. Set it as the `OPENROUTER_API_KEY` environment variable for your deployment.
+## Key Features
 
-| Model | Best for |
-|---|---|
-| `deepseek/deepseek-v4-flash` (default) | Best cost/accuracy balance for most uses |
+- **Multi-Tenant Workspaces & RBAC**: Isolate connections, dashboards, and knowledge bases per workspace. Assign Owner, Admin, or Member roles with fine-grained capability checks.
+- **OpenRouter AI Engine**: Powered by `deepseek/deepseek-v4-flash` via OpenRouter.
+- **Interactive Dashboards & Charts**: Automated ECharts visual inference (Bar, Line, Area, Pie, Number, Table) with query panel customization.
+- **Semantic Layer**: Define custom business metrics, explicit join paths, synonyms, and value mappings to guide AI SQL generation.
+- **Safety & Defense in Depth**: Read-only AST validation, SSRF protection, host allowlisting, and 5s statement timeouts.
+- **PostgreSQL Persistence**: All state (users, workspaces, connections encrypted at rest, query history, dashboards, verified Q&A) is persisted asynchronously in PostgreSQL.
 
-**Privacy:** what's sent to the AI to generate SQL is your question plus the
-context BoloDB builds around it: the database *structure* (table/column names,
-keys), a few sample values and sample rows per table, your confirmed
-business-term glossary, previously verified question→SQL examples, and the
-last couple of conversation turns. **The prompt never includes** bulk table
-contents, query results, or credentials — the OpenRouter API key travels only in
-the request's authentication header. See
-[docs/03-the-ai-layer-openrouter.md](docs/03-the-ai-layer-openrouter.md) for exactly
-what's in every prompt.
+---
 
-## Database connection strings
+## Supported Database Connections
 
-| Database | Format |
+| Database | Connection URL Format |
 |---|---|
-| SQLite | `sqlite:///C:/path/to/file.db` |
 | PostgreSQL | `postgresql://user:pass@host:5432/dbname` |
 | MySQL | `mysql+pymysql://user:pass@host:3306/dbname` |
+| SQLite | `sqlite:///C:/path/to/file.db` |
 | SQL Server | `mssql+pyodbc://user:pass@server/db?driver=ODBC+Driver+17+for+SQL+Server` |
+| DuckDB | `duckdb:///path/to/file.duckdb` |
+
+---
+
+## Architecture & Code Map
+
+Full file and directory index is available in [`docs/07-file-map.md`](docs/07-file-map.md).
+
+```text
+ Browser (SvelteKit 5 Frontend, frontend/src)
+    │  HTTP / SSE streaming (JWT cookie auth, X-Workspace-Id and X-Db-Id headers)
+    ▼
+ FastAPI Backend (backend/app)
+    ├── controllers/query.py ─── The query pipeline (knowledge → schema link → LLM → repair)
+    ├── llm.py ───────────────── OpenRouter provider (deepseek/deepseek-v4-flash)
+    ├── schema_link.py ───────── Schema linking, budget management & table scoring
+    ├── sqlvalidate.py ───────── AST static SQL validation
+    ├── repair.py ────────────── Self-repair loop for auto-correcting broken SQL
+    ├── semantic.py ──────────── Semantic layer catalog & inference
+    ├── database.py ──────────── DB introspection & read-only execution guard
+    └── pgdatabase/ ──────────── Async PostgreSQL persistence (KnowledgeService, Users, Workspaces, Dashboards)
+```
 
-Tip: connect with a **read-only database account** for safety (BoloDB also
-enforces read-only itself — see
-[docs/05-safety-validation-and-self-repair.md](docs/05-safety-validation-and-self-repair.md)).
-
-## How it works
-
-1. **Connect** a database and ensure your OpenRouter API key is set in the environment.
-2. **Onboard** (first time) — BoloDB profiles the tables, confirms business-term meanings with you, and runs a few starter questions for you to verify
-3. **Ask** questions in plain English. Every answer includes:
-   - A plain-English restatement ("I summed completed orders for this month")
-   - A confidence level (High/Medium/Low) based on real signals
-   - The results table and SQL on a toggle
-   - Automatic self-repair: generated SQL is validated against your schema and fixed before you ever see an error
-4. **Verify** — click "Yes, correct" to save the answer. Similar future questions reuse it and show higher confidence. The trust level climbs from Supervised to Assisted to Trusted
-
-The full pipeline, step by step with code pointers:
-[docs/02-how-a-question-becomes-an-answer.md](docs/02-how-a-question-becomes-an-answer.md).
+---
 
-## Useful Docker Commands
+## Running Unit Tests
 
 ```bash
-docker compose up -d           # Start the application in the background
-docker compose logs -f         # View live logs from all services
-docker compose down            # Stop all services
-docker compose build --no-cache # Rebuild all images from scratch
+pip install -r backend/requirements.txt
+pytest tests -v
 ```
 
-## Running tests
+The test suite runs entirely offline using mock OpenRouter providers.
 
-```bash
-pip install -r backend/requirements.txt
-pytest tests
-```
+---
 
-The test suite needs no network and no API key — all AI calls are faked.
+## Privacy & Security
 
-## Privacy
+- Connection credentials are **encrypted at rest** using Fernet symmetric encryption key (`RECENT_CONNECTIONS_KEY`).
+- AI prompts include only database structure, compact column schemas, verified Q&A examples, and business glossary definitions. Bulk data rows, query results, and credentials are **never sent** to the AI model.
+- All SQL execution runs in strict **read-only** mode with statement timeout protection.
 
-- All learned knowledge and user settings are stored locally (`~/.bolodb/`) and in the local MongoDB container volume. The API key is read from the environment and never stored on disk.
-- To generate SQL, the AI is sent your question plus context: the schema, a few sample values/rows per table, your confirmed glossary terms, verified question→SQL examples, and recent conversation turns. The prompt never includes bulk table data, query results, or credentials (the API key is used only as the request's authentication header).
-- Queries run strictly read-only.
-- No telemetry, no cloud sync.
+---
 
 ## License
 
-MIT — see [LICENSE](LICENSE).
+MIT — see [LICENSE](LICENSE).

backend/DOCKERFILE

@@ -20,9 +20,12 @@ RUN pip install --no-cache-dir -r /app/backend/requirements.txt
 COPY backend/        /app/backend/
 
 # ── 3. Create non-root user ──
+# /app/data holds the regenerable sample database; create it owned by the
+# runtime user so it is writable whether or not a volume is mounted over it.
 RUN addgroup --system bolodb && adduser --system --home /home/bolodb --ingroup bolodb bolodb \
-    && mkdir -p /home/bolodb/.bolodb \
-    && chown -R bolodb:bolodb /home/bolodb/.bolodb /app
+    && mkdir -p /app/data \
+    && chmod +x /app/backend/docker-entrypoint.sh \
+    && chown -R bolodb:bolodb /app
 
 USER bolodb
 
@@ -31,5 +34,9 @@ EXPOSE 4321
 HEALTHCHECK --interval=30s --timeout=5s --retries=3 \
     CMD python -c "import httpx; httpx.get('http://localhost:4321/api/health', timeout=3).raise_for_status()" || exit 1
 
+# Migrations run first, so a deploy pointed at a new or stale database brings
+# the schema to head before the app serves its first request.
+ENTRYPOINT ["/app/backend/docker-entrypoint.sh"]
+
 CMD ["python", "-m", "uvicorn", "backend.app.server:create_app", \
      "--factory", "--host", "0.0.0.0", "--port", "4321", "--log-level", "warning"]

backend/alembic/env.py

@@ -39,9 +39,23 @@ def run_migrations_offline() -> None:
         context.run_migrations()
 
 
+# Any 64-bit constant works; it only has to be the same in every process that
+# migrates this database. Derived from the project name so it cannot collide
+# with an advisory lock another application takes on a shared cluster.
+_MIGRATION_LOCK_ID = 8410231166942030001
+
+
 def do_run_migrations(connection):
     context.configure(connection=connection, target_metadata=target_metadata)
     with context.begin_transaction():
+        # Containers start migrations concurrently on a scaled deployment, and
+        # two Alembic runs against one database race on alembic_version. The
+        # transaction-scoped advisory lock makes the second wait for the first
+        # and then find nothing left to apply; it is released on commit or
+        # rollback, so a crashed migration cannot wedge later deploys.
+        connection.exec_driver_sql(
+            f"SELECT pg_advisory_xact_lock({_MIGRATION_LOCK_ID})"
+        )
         context.run_migrations()