Add safety intelligence schemas and refinery RAG knowledge base

.gitignore

@@ -6,6 +6,16 @@ __pycache__/
 *.sqlite3
 *.db
 
+# RAG indexes are generated locally from authorized documents and should not
+# be committed to the public repository.
+backend/data/knowledge/vector_store/*
+!backend/data/knowledge/vector_store/.gitkeep
+backend/data/knowledge/processed/*
+!backend/data/knowledge/processed/.gitkeep
+backend/data/knowledge/model_cache/
+.cache/
+.huggingface/
+
 # Virtual Environment
 .venv/
 venv/
@@ -24,4 +34,4 @@ node_modules/
 
 # OS
 .DS_Store
-Thumbs.db
+Thumbs.db

AGENTS.md

@@ -0,0 +1,9 @@
+# Repository Notes
+
+- The FastAPI application and Pydantic contracts live under `backend/app`.
+- Install `requirements-dev.txt` and run backend tests with `.venv/bin/python -m pytest`.
+- Import public contracts from `app.schemas`; keep internal schema imports absolute.
+- Do not push directly to `main`.
+- Do not rename existing API fields or `maintainance.py` without checking every usage.
+- Keep safety recommendations advisory and human-approved by default.
+- Update schema tests, `docs/examples`, and `docs/SCHEMA_ARCHITECTURE.md` together.

README.md

@@ -1,12 +1,117 @@
-# SpecGuard
+# Digital Twin Petroleum Refinery Simulator
 
+A modular, realistic Digital Twin simulator of a petroleum refinery process unit. This simulator generates continuous, time-series multivariate industrial sensor streams with physically meaningful dependencies between sensors. It is designed specifically to serve as the data source for training and validating AI-powered Industrial Safety Intelligence platforms to detect both simple faults and complex, compound risks.
 
-nahi voting chal raha hai: 
-1 -4 (decided)
-8-2 
-5-1 
-7-1 
-toh ab first wala karenge
+## Features
 
-pakka first wala?
-ek baar research karte hai fir dedcide karte hai!
+- **Realistic Multivariate Dependencies**: Uses statistical properties extracted from the Tennessee Eastman Process (TEP) to ensure that variables (like pump speed, pipeline pressure, and flow rates) are realistically cross-correlated.
+- **Diverse Simulation Scenarios**: Simulates normal refinery operations as well as 6 distinct fault scenarios ranging from minor equipment wear to severe compound explosion risks.
+- **Granular Event Tracking**: Tracks not just physical sensors (SCADA and Gas), but also worker movements, Permit-to-Work (PTW) statuses, scheduled/reactive maintenance, synthesized CCTV events, and realistic operator shift logs.
+- **Flexible Exporting**: Outputs telemetry as a single monolithic JSON Lines (JSONL) stream or as split relational CSV files by data category.
+
+## Project Structure
+
+The repository is organized into a modular architecture:
+
+```text
+digital_twin/
+├── simulator/
+│   ├── config.py              # Global configuration, zone definitions, equipment setup
+│   ├── clock.py               # Simulation clock (tick manager)
+│   ├── plant.py               # Plant orchestrator coupling all models together
+│   │
+│   ├── equipment/             # Physical equipment models
+│   │   ├── storage_tank.py
+│   │   ├── pipeline.py
+│   │   ├── pump.py
+│   │   ├── valve.py
+│   │   └── ventilation.py
+│   │
+│   ├── sensor_models/         # Sensor generation and noise
+│   │   ├── process_model.py   # Core TEP-derived multivariate process model
+│   │   ├── scada_sensors.py   # Pressure, temperature, flow, speed SCADA readings
+│   │   ├── gas_sensors.py     # HC (LEL), H2S, VOC, and O2 sensors
+│   │   └── noise.py           # Gaussian noise, drift, and dropouts
+│   │
+│   ├── events/                # Human and organizational event models
+│   │   ├── worker_events.py   # Worker location, task assignment, PPE tracking
+│   │   ├── permit_to_work.py  # PTW lifecycle (Hot Work, Confined Space, etc.)
+│   │   ├── maintenance.py     # Maintenance activities and equipment isolation
+│   │   ├── shift_logs.py      # Automated shift log generation
+│   │   └── cctv_events.py     # Synthesized structured CCTV detections
+│   │
+│   ├── scenario_engine/       # Fault injection and scenario progression
+│   │   ├── base_scenario.py
+│   │   ├── normal.py
+│   │   ├── gas_leak.py
+│   │   ├── ventilation_failure.py
+│   │   ├── pump_failure.py
+│   │   ├── hot_work_gas_leak.py
+│   │   ├── confined_space.py
+│   │   └── explosion_risk.py
+│   │
+│   ├── export/                # Exporters
+│   │   ├── csv_exporter.py
+│   │   └── json_exporter.py
+│   │
+│   └── tep/                   # Tennessee Eastman Process references
+│       ├── extract_statistics.py
+│       └── tep_statistics.json
+│
+├── simulate.py                # Main CLI entry point
+├── requirements.txt           # Project dependencies
+└── output/                    # Generated simulation data directory
+```
+
+## Available Scenarios
+
+1. **`normal`** (100,000 rows): Normal steady-state operation with natural variability, diurnal cycles, and shift changes.
+2. **`gas_leak`** (5,000 rows): Small gas leak developing from pump seal degradation, leading to detection and emergency response.
+3. **`ventilation_failure`** (5,000 rows): Fan motor degradation leading to complete failure and gas accumulation.
+4. **`pump_failure`** (5,000 rows): Bearing wear progression causing vibration, overheating, seizure, and switchover.
+5. **`hot_work_gas_leak`** (5,000 rows): Compound scenario where an undetected leak develops near an active hot work permit.
+6. **`confined_space`** (5,000 rows): O2 depletion in a confined space, worker entry without proper gas testing, and rescue.
+7. **`explosion_risk`** (5,000 rows): Maximum risk compound scenario featuring a simultaneous pump seal failure, ventilation failure, and active hot work leading to an Emergency Shut Down (ESD).
+
+## Installation
+
+Ensure you have a Python environment (e.g., conda) setup. Install the dependencies:
+
+```bash
+pip install -r requirements.txt
+```
+
+*(Note: The `numpy`, `pandas`, and `scipy` packages are required.)*
+
+## Usage
+
+Use the `simulate.py` CLI to run simulations. 
+
+### Basic Usage
+
+Run the normal scenario (default 100,000 seconds/rows) and output to CSV:
+```bash
+python simulate.py --scenario normal
+```
+
+Run a specific fault scenario:
+```bash
+python simulate.py --scenario gas_leak
+```
+
+### Advanced Usage
+
+Run all scenarios sequentially to generate the complete 130,000-row dataset, outputting both JSON and split CSV files:
+```bash
+python simulate.py --scenario all --format both --split --output ./output
+```
+
+Override the duration of a scenario (in seconds/ticks):
+```bash
+python simulate.py --scenario ventilation_failure --duration 7200
+```
+
+### Output Formats
+- **Combined CSV**: A monolithic `simulation_data.csv` containing all 56+ telemetry columns.
+- **Split CSVs (`--split`)**: Separates data into domain-specific files (`scada.csv`, `gas.csv`, `workers.csv`, `permits.csv`, `maintenance.csv`, `equipment.csv`, `shift_logs.csv`, `cctv.csv`). Recommended for building relational databases.
+- **JSON Lines (`--format json`)**: A hierarchical `simulation_data.jsonl` where each line is a tick containing nested telemetry categories. Highly recommended for direct ingestion into document databases or stream processing tools.

backend/app/rag/__init__.py

@@ -0,0 +1,32 @@
+"""Document ingestion and evidence retrieval primitives for SpecGuard."""
+
+from app.rag.chunker import chunk_pages
+from app.rag.embedder import DeterministicEmbedder, SentenceTransformerEmbedder
+from app.rag.metadata import enrich_chunks, load_manifest
+from app.rag.models import (
+    DocumentChunk,
+    DocumentPage,
+    ManifestEntry,
+    RetrievalQuery,
+    RetrievalResult,
+    SourceDocument,
+)
+from app.rag.retriever import Retriever, risk_to_retrieval_query
+from app.rag.vector_store import JsonVectorStore
+
+__all__ = [
+    "DeterministicEmbedder",
+    "DocumentChunk",
+    "DocumentPage",
+    "JsonVectorStore",
+    "ManifestEntry",
+    "RetrievalQuery",
+    "RetrievalResult",
+    "Retriever",
+    "SentenceTransformerEmbedder",
+    "SourceDocument",
+    "chunk_pages",
+    "enrich_chunks",
+    "load_manifest",
+    "risk_to_retrieval_query",
+]

backend/app/rag/chunker.py

@@ -0,0 +1,140 @@
+"""Deterministic, section-aware chunking for extracted document pages."""
+
+from __future__ import annotations
+
+import hashlib
+import re
+from dataclasses import dataclass
+
+from app.rag.models import DocumentChunk, DocumentPage
+
+
+DEFAULT_MAX_TOKENS = 700
+DEFAULT_OVERLAP_TOKENS = 120
+_HEADING = re.compile(r"^(?:#{1,6}\s+|\d+(?:\.\d+)*\s+)(.+?)\s*$")
+
+
+@dataclass(frozen=True)
+class _Unit:
+    words: tuple[str, ...]
+    page_number: int
+    section: str | None
+
+
+def _section_name(line: str) -> str | None:
+    match = _HEADING.match(line.strip())
+    if not match:
+        return None
+    value = match.group(1).strip().strip("#").strip()
+    return value or None
+
+
+def _units(pages: list[DocumentPage]) -> list[_Unit]:
+    result: list[_Unit] = []
+    section: str | None = None
+    for page in pages:
+        paragraph: list[str] = []
+
+        def flush() -> None:
+            if paragraph:
+                result.append(_Unit(tuple(" ".join(paragraph).split()), page.page_number, section))
+                paragraph.clear()
+
+        for line in page.text.splitlines():
+            heading = _section_name(line)
+            if heading:
+                flush()
+                section = heading
+                continue
+            if not line.strip():
+                flush()
+                continue
+            paragraph.append(line.strip())
+        flush()
+    return result
+
+
+def _chunk_id(document_id: str, index: int, text: str) -> str:
+    digest = hashlib.sha1(f"{document_id}:{index}:{text}".encode("utf-8")).hexdigest()[:16]
+    return f"{document_id}-chunk-{index:04d}-{digest}"
+
+
+def chunk_pages(
+    pages: list[DocumentPage],
+    *,
+    max_tokens: int = DEFAULT_MAX_TOKENS,
+    overlap_tokens: int = DEFAULT_OVERLAP_TOKENS,
+) -> list[DocumentChunk]:
+    """Chunk pages using word counts as a transparent token approximation.
+
+    A tokenizer is intentionally not required for the prototype. The overlap
+    is carried from the previous chunk so a safety condition split at a boundary
+    remains visible to the next retrieval result.
+    """
+
+    if max_tokens < 1:
+        raise ValueError("max_tokens must be positive")
+    if overlap_tokens < 0 or overlap_tokens >= max_tokens:
+        raise ValueError("overlap_tokens must be between zero and max_tokens - 1")
+    if not pages:
+        return []
+    document_ids = {page.document_id for page in pages}
+    if len(document_ids) != 1:
+        raise ValueError("chunk_pages accepts pages from one document at a time")
+
+    units = _units(pages)
+    if not units:
+        return []
+    document = pages[0]
+    chunks: list[DocumentChunk] = []
+    current: list[_Unit] = []
+    current_words = 0
+
+    def emit(items: list[_Unit]) -> None:
+        if not items:
+            return
+        text = " ".join(word for item in items for word in item.words).strip()
+        if len(text.split()) < 3 or len(text) < 10:
+            return
+        start = min(item.page_number for item in items)
+        end = max(item.page_number for item in items)
+        sections = [item.section for item in items if item.section]
+        # When a short document fits in one chunk, retain its first section as
+        # a useful navigation hint even if later sections are also present.
+        section = sections[0] if sections else None
+        index = len(chunks)
+        chunks.append(
+            DocumentChunk(
+                chunk_id=_chunk_id(document.document_id, index, text),
+                document_id=document.document_id,
+                document_title=document.source_title,
+                text=text,
+                page_start=start,
+                page_end=end,
+                section=section,
+                document_type=document.document_type,
+                authority=document.authority,
+                is_synthetic=document.is_synthetic,
+                source_url=document.source_url,
+                source_path=document.source_path,
+                publication_date=document.publication_date,
+                version=document.version,
+                tags=document.tags,
+            )
+        )
+
+    for unit in units:
+        words = list(unit.words)
+        while words:
+            available = max_tokens - current_words
+            take = min(len(words), available)
+            current.append(_Unit(tuple(words[:take]), unit.page_number, unit.section))
+            current_words += take
+            words = words[take:]
+            if current_words >= max_tokens:
+                emit(current)
+                overlap = [word for item in current for word in item.words][-overlap_tokens:]
+                current = [_Unit(tuple(overlap), current[-1].page_number, current[-1].section)] if overlap else []
+                current_words = len(overlap)
+    emit(current)
+    return chunks