diff --git a/.gitignore b/.gitignore index c474155..816cb4c 100644 --- a/.gitignore +++ b/.gitignore @@ -6,6 +6,16 @@ __pycache__/ *.sqlite3 *.db +# RAG indexes are generated locally from authorized documents and should not +# be committed to the public repository. +backend/data/knowledge/vector_store/* +!backend/data/knowledge/vector_store/.gitkeep +backend/data/knowledge/processed/* +!backend/data/knowledge/processed/.gitkeep +backend/data/knowledge/model_cache/ +.cache/ +.huggingface/ + # Virtual Environment .venv/ venv/ @@ -24,4 +34,4 @@ node_modules/ # OS .DS_Store -Thumbs.db \ No newline at end of file +Thumbs.db diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..d8436cf --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,9 @@ +# Repository Notes + +- The FastAPI application and Pydantic contracts live under `backend/app`. +- Install `requirements-dev.txt` and run backend tests with `.venv/bin/python -m pytest`. +- Import public contracts from `app.schemas`; keep internal schema imports absolute. +- Do not push directly to `main`. +- Do not rename existing API fields or `maintainance.py` without checking every usage. +- Keep safety recommendations advisory and human-approved by default. +- Update schema tests, `docs/examples`, and `docs/SCHEMA_ARCHITECTURE.md` together. diff --git a/README.md b/README.md index efdf557..6e36477 100644 --- a/README.md +++ b/README.md @@ -1,12 +1,117 @@ -# SpecGuard +# Digital Twin Petroleum Refinery Simulator +A modular, realistic Digital Twin simulator of a petroleum refinery process unit. This simulator generates continuous, time-series multivariate industrial sensor streams with physically meaningful dependencies between sensors. It is designed specifically to serve as the data source for training and validating AI-powered Industrial Safety Intelligence platforms to detect both simple faults and complex, compound risks. -nahi voting chal raha hai: -1 -4 (decided) -8-2 -5-1 -7-1 -toh ab first wala karenge +## Features -pakka first wala? -ek baar research karte hai fir dedcide karte hai! +- **Realistic Multivariate Dependencies**: Uses statistical properties extracted from the Tennessee Eastman Process (TEP) to ensure that variables (like pump speed, pipeline pressure, and flow rates) are realistically cross-correlated. +- **Diverse Simulation Scenarios**: Simulates normal refinery operations as well as 6 distinct fault scenarios ranging from minor equipment wear to severe compound explosion risks. +- **Granular Event Tracking**: Tracks not just physical sensors (SCADA and Gas), but also worker movements, Permit-to-Work (PTW) statuses, scheduled/reactive maintenance, synthesized CCTV events, and realistic operator shift logs. +- **Flexible Exporting**: Outputs telemetry as a single monolithic JSON Lines (JSONL) stream or as split relational CSV files by data category. + +## Project Structure + +The repository is organized into a modular architecture: + +```text +digital_twin/ +├── simulator/ +│ ├── config.py # Global configuration, zone definitions, equipment setup +│ ├── clock.py # Simulation clock (tick manager) +│ ├── plant.py # Plant orchestrator coupling all models together +│ │ +│ ├── equipment/ # Physical equipment models +│ │ ├── storage_tank.py +│ │ ├── pipeline.py +│ │ ├── pump.py +│ │ ├── valve.py +│ │ └── ventilation.py +│ │ +│ ├── sensor_models/ # Sensor generation and noise +│ │ ├── process_model.py # Core TEP-derived multivariate process model +│ │ ├── scada_sensors.py # Pressure, temperature, flow, speed SCADA readings +│ │ ├── gas_sensors.py # HC (LEL), H2S, VOC, and O2 sensors +│ │ └── noise.py # Gaussian noise, drift, and dropouts +│ │ +│ ├── events/ # Human and organizational event models +│ │ ├── worker_events.py # Worker location, task assignment, PPE tracking +│ │ ├── permit_to_work.py # PTW lifecycle (Hot Work, Confined Space, etc.) +│ │ ├── maintenance.py # Maintenance activities and equipment isolation +│ │ ├── shift_logs.py # Automated shift log generation +│ │ └── cctv_events.py # Synthesized structured CCTV detections +│ │ +│ ├── scenario_engine/ # Fault injection and scenario progression +│ │ ├── base_scenario.py +│ │ ├── normal.py +│ │ ├── gas_leak.py +│ │ ├── ventilation_failure.py +│ │ ├── pump_failure.py +│ │ ├── hot_work_gas_leak.py +│ │ ├── confined_space.py +│ │ └── explosion_risk.py +│ │ +│ ├── export/ # Exporters +│ │ ├── csv_exporter.py +│ │ └── json_exporter.py +│ │ +│ └── tep/ # Tennessee Eastman Process references +│ ├── extract_statistics.py +│ └── tep_statistics.json +│ +├── simulate.py # Main CLI entry point +├── requirements.txt # Project dependencies +└── output/ # Generated simulation data directory +``` + +## Available Scenarios + +1. **`normal`** (100,000 rows): Normal steady-state operation with natural variability, diurnal cycles, and shift changes. +2. **`gas_leak`** (5,000 rows): Small gas leak developing from pump seal degradation, leading to detection and emergency response. +3. **`ventilation_failure`** (5,000 rows): Fan motor degradation leading to complete failure and gas accumulation. +4. **`pump_failure`** (5,000 rows): Bearing wear progression causing vibration, overheating, seizure, and switchover. +5. **`hot_work_gas_leak`** (5,000 rows): Compound scenario where an undetected leak develops near an active hot work permit. +6. **`confined_space`** (5,000 rows): O2 depletion in a confined space, worker entry without proper gas testing, and rescue. +7. **`explosion_risk`** (5,000 rows): Maximum risk compound scenario featuring a simultaneous pump seal failure, ventilation failure, and active hot work leading to an Emergency Shut Down (ESD). + +## Installation + +Ensure you have a Python environment (e.g., conda) setup. Install the dependencies: + +```bash +pip install -r requirements.txt +``` + +*(Note: The `numpy`, `pandas`, and `scipy` packages are required.)* + +## Usage + +Use the `simulate.py` CLI to run simulations. + +### Basic Usage + +Run the normal scenario (default 100,000 seconds/rows) and output to CSV: +```bash +python simulate.py --scenario normal +``` + +Run a specific fault scenario: +```bash +python simulate.py --scenario gas_leak +``` + +### Advanced Usage + +Run all scenarios sequentially to generate the complete 130,000-row dataset, outputting both JSON and split CSV files: +```bash +python simulate.py --scenario all --format both --split --output ./output +``` + +Override the duration of a scenario (in seconds/ticks): +```bash +python simulate.py --scenario ventilation_failure --duration 7200 +``` + +### Output Formats +- **Combined CSV**: A monolithic `simulation_data.csv` containing all 56+ telemetry columns. +- **Split CSVs (`--split`)**: Separates data into domain-specific files (`scada.csv`, `gas.csv`, `workers.csv`, `permits.csv`, `maintenance.csv`, `equipment.csv`, `shift_logs.csv`, `cctv.csv`). Recommended for building relational databases. +- **JSON Lines (`--format json`)**: A hierarchical `simulation_data.jsonl` where each line is a tick containing nested telemetry categories. Highly recommended for direct ingestion into document databases or stream processing tools. diff --git a/backend/app/rag/__init__.py b/backend/app/rag/__init__.py new file mode 100644 index 0000000..bf72915 --- /dev/null +++ b/backend/app/rag/__init__.py @@ -0,0 +1,32 @@ +"""Document ingestion and evidence retrieval primitives for SpecGuard.""" + +from app.rag.chunker import chunk_pages +from app.rag.embedder import DeterministicEmbedder, SentenceTransformerEmbedder +from app.rag.metadata import enrich_chunks, load_manifest +from app.rag.models import ( + DocumentChunk, + DocumentPage, + ManifestEntry, + RetrievalQuery, + RetrievalResult, + SourceDocument, +) +from app.rag.retriever import Retriever, risk_to_retrieval_query +from app.rag.vector_store import JsonVectorStore + +__all__ = [ + "DeterministicEmbedder", + "DocumentChunk", + "DocumentPage", + "JsonVectorStore", + "ManifestEntry", + "RetrievalQuery", + "RetrievalResult", + "Retriever", + "SentenceTransformerEmbedder", + "SourceDocument", + "chunk_pages", + "enrich_chunks", + "load_manifest", + "risk_to_retrieval_query", +] diff --git a/backend/app/rag/chunker.py b/backend/app/rag/chunker.py new file mode 100644 index 0000000..24344cf --- /dev/null +++ b/backend/app/rag/chunker.py @@ -0,0 +1,140 @@ +"""Deterministic, section-aware chunking for extracted document pages.""" + +from __future__ import annotations + +import hashlib +import re +from dataclasses import dataclass + +from app.rag.models import DocumentChunk, DocumentPage + + +DEFAULT_MAX_TOKENS = 700 +DEFAULT_OVERLAP_TOKENS = 120 +_HEADING = re.compile(r"^(?:#{1,6}\s+|\d+(?:\.\d+)*\s+)(.+?)\s*$") + + +@dataclass(frozen=True) +class _Unit: + words: tuple[str, ...] + page_number: int + section: str | None + + +def _section_name(line: str) -> str | None: + match = _HEADING.match(line.strip()) + if not match: + return None + value = match.group(1).strip().strip("#").strip() + return value or None + + +def _units(pages: list[DocumentPage]) -> list[_Unit]: + result: list[_Unit] = [] + section: str | None = None + for page in pages: + paragraph: list[str] = [] + + def flush() -> None: + if paragraph: + result.append(_Unit(tuple(" ".join(paragraph).split()), page.page_number, section)) + paragraph.clear() + + for line in page.text.splitlines(): + heading = _section_name(line) + if heading: + flush() + section = heading + continue + if not line.strip(): + flush() + continue + paragraph.append(line.strip()) + flush() + return result + + +def _chunk_id(document_id: str, index: int, text: str) -> str: + digest = hashlib.sha1(f"{document_id}:{index}:{text}".encode("utf-8")).hexdigest()[:16] + return f"{document_id}-chunk-{index:04d}-{digest}" + + +def chunk_pages( + pages: list[DocumentPage], + *, + max_tokens: int = DEFAULT_MAX_TOKENS, + overlap_tokens: int = DEFAULT_OVERLAP_TOKENS, +) -> list[DocumentChunk]: + """Chunk pages using word counts as a transparent token approximation. + + A tokenizer is intentionally not required for the prototype. The overlap + is carried from the previous chunk so a safety condition split at a boundary + remains visible to the next retrieval result. + """ + + if max_tokens < 1: + raise ValueError("max_tokens must be positive") + if overlap_tokens < 0 or overlap_tokens >= max_tokens: + raise ValueError("overlap_tokens must be between zero and max_tokens - 1") + if not pages: + return [] + document_ids = {page.document_id for page in pages} + if len(document_ids) != 1: + raise ValueError("chunk_pages accepts pages from one document at a time") + + units = _units(pages) + if not units: + return [] + document = pages[0] + chunks: list[DocumentChunk] = [] + current: list[_Unit] = [] + current_words = 0 + + def emit(items: list[_Unit]) -> None: + if not items: + return + text = " ".join(word for item in items for word in item.words).strip() + if len(text.split()) < 3 or len(text) < 10: + return + start = min(item.page_number for item in items) + end = max(item.page_number for item in items) + sections = [item.section for item in items if item.section] + # When a short document fits in one chunk, retain its first section as + # a useful navigation hint even if later sections are also present. + section = sections[0] if sections else None + index = len(chunks) + chunks.append( + DocumentChunk( + chunk_id=_chunk_id(document.document_id, index, text), + document_id=document.document_id, + document_title=document.source_title, + text=text, + page_start=start, + page_end=end, + section=section, + document_type=document.document_type, + authority=document.authority, + is_synthetic=document.is_synthetic, + source_url=document.source_url, + source_path=document.source_path, + publication_date=document.publication_date, + version=document.version, + tags=document.tags, + ) + ) + + for unit in units: + words = list(unit.words) + while words: + available = max_tokens - current_words + take = min(len(words), available) + current.append(_Unit(tuple(words[:take]), unit.page_number, unit.section)) + current_words += take + words = words[take:] + if current_words >= max_tokens: + emit(current) + overlap = [word for item in current for word in item.words][-overlap_tokens:] + current = [_Unit(tuple(overlap), current[-1].page_number, current[-1].section)] if overlap else [] + current_words = len(overlap) + emit(current) + return chunks diff --git a/backend/app/rag/cli.py b/backend/app/rag/cli.py new file mode 100644 index 0000000..8b6dd2a --- /dev/null +++ b/backend/app/rag/cli.py @@ -0,0 +1,246 @@ +"""Command-line entry point for local knowledge-base ingestion and queries.""" + +from __future__ import annotations + +import argparse +from dataclasses import dataclass, field +from pathlib import Path + +from app.rag.chunker import chunk_pages +from app.rag.document_loader import DocumentLoadError, load_document +from app.rag.embedder import DeterministicEmbedder, Embedder, SentenceTransformerEmbedder +from app.rag.metadata import enrich_chunks, load_manifest +from app.rag.models import RetrievalQuery +from app.rag.retriever import Retriever +from app.rag.text_cleaner import clean_pages +from app.rag.vector_store import JsonVectorStore +from app.schemas.common import HazardCode, PermitType, RiskType + + +REPO_ROOT = Path(__file__).resolve().parents[3] +KNOWLEDGE_ROOT = REPO_ROOT / "backend/data/knowledge" +DEFAULT_MANIFEST = KNOWLEDGE_ROOT / "manifests/documents.json" +DEFAULT_STORE = KNOWLEDGE_ROOT / "vector_store/chunks.json" + + +@dataclass +class IngestionSummary: + """Counts and diagnostics emitted by one deterministic ingestion run.""" + + documents_total: int = 0 + documents_loaded: int = 0 + documents_skipped: int = 0 + missing_files: int = 0 + extraction_failures: int = 0 + empty_extractions: int = 0 + chunks_created: int = 0 + chunks_indexed: int = 0 + persisted_chunks: int = 0 + skipped_messages: list[str] = field(default_factory=list) + + +def _embedder(name: str) -> Embedder: + if name == "deterministic": + return DeterministicEmbedder() + if name == "sentence-transformers": + return SentenceTransformerEmbedder() + raise ValueError("embedder must be deterministic or sentence-transformers") + + +def _source_root(manifest_path: Path) -> Path: + """Resolve manifest paths such as ``raw/regulations/file.pdf``.""" + + return manifest_path.resolve().parent.parent + + +def ingest_documents( + manifest_path: Path, + store_path: Path, + *, + rebuild: bool, + embedder_name: str, +) -> IngestionSummary: + entries = load_manifest(manifest_path) + summary = IngestionSummary(documents_total=len(entries)) + chunks = [] + processed_document_ids: list[str] = [] + + for entry in entries: + if not entry.local_path: + summary.skipped_messages.append(f"{entry.document_id}: no local_path in manifest") + continue + try: + loaded = load_document(entry.to_source_document(), base_dir=_source_root(manifest_path)) + except DocumentLoadError as exc: + message = f"{entry.document_id}: {exc}" + summary.skipped_messages.append(message) + if "does not exist" in str(exc): + summary.missing_files += 1 + elif "contains no text" in str(exc) or "no extractable" in str(exc): + summary.empty_extractions += 1 + summary.extraction_failures += 1 + else: + summary.extraction_failures += 1 + continue + + if not loaded: + summary.empty_extractions += 1 + summary.extraction_failures += 1 + summary.skipped_messages.append( + f"{entry.document_id}: no extractable text; scanned PDFs need manual OCR review" + ) + continue + + summary.documents_loaded += 1 + processed_document_ids.append(entry.document_id) + cleaned = clean_pages(loaded) + chunks.extend(enrich_chunks(chunk_pages(cleaned))) + + summary.chunks_created = len(chunks) + embedder = _embedder(embedder_name) + embeddings = embedder.embed([chunk.text for chunk in chunks]) + store = JsonVectorStore(store_path) + if rebuild: + store.rebuild(chunks, embeddings) + else: + # Replace successfully reprocessed documents so repeated ingestion is + # idempotent even if a document's chunk boundaries have changed. + store.delete_documents(processed_document_ids) + store.add(chunks, embeddings) + summary.documents_skipped = len(summary.skipped_messages) + summary.chunks_indexed = len(chunks) + summary.persisted_chunks = store.count + return summary + + +def _print_summary(summary: IngestionSummary) -> None: + print("Ingestion summary:") + print(f" Documents loaded: {summary.documents_loaded}/{summary.documents_total}") + print(f" Documents skipped: {summary.documents_skipped}") + print(f" Missing files: {summary.missing_files}") + print(f" Extraction failures: {summary.extraction_failures}") + print(f" Chunks created: {summary.chunks_created}") + print(f" Chunks indexed: {summary.chunks_indexed}") + print(f" Persisted chunks: {summary.persisted_chunks}") + for message in summary.skipped_messages: + print(f" Skipped: {message}") + + +def ingest(manifest_path: Path, store_path: Path, *, rebuild: bool, embedder_name: str) -> int: + _print_summary( + ingest_documents( + manifest_path, + store_path, + rebuild=rebuild, + embedder_name=embedder_name, + ) + ) + return 0 + + +def inspect(manifest_path: Path, store_path: Path) -> int: + entries = load_manifest(manifest_path) + store = JsonVectorStore(store_path) + print(f"Manifest entries: {len(entries)}") + print(f"Persisted chunks: {store.count}") + print(f"Embedding dimension: {store.embedding_dimension or 'none'}") + for entry in entries: + print(f"- {entry.document_id}: {entry.processing_status} ({entry.local_path or 'not added'})") + return 0 + + +def query( + text: str, + store_path: Path, + *, + top_k: int, + mode: str, + embedder_name: str, + risk_types: list[str] | None = None, + hazard_codes: list[str] | None = None, + permit_types: list[str] | None = None, + document_types: list[str] | None = None, +) -> int: + retrieval_query = RetrievalQuery( + query_text=text, + top_k=top_k, + risk_types=[RiskType(value) for value in risk_types or []], + hazard_codes=[HazardCode(value) for value in hazard_codes or []], + permit_types=[PermitType(value) for value in permit_types or []], + document_types=document_types or [], + ) + retriever = Retriever(JsonVectorStore(store_path), _embedder(embedder_name)) + results = retriever.retrieve(retrieval_query, mode=mode) + if not results: + print("No matching evidence found.") + return 0 + for result in results: + pages = ( + f"p. {result.page_start}" + if result.page_start == result.page_end + else f"pp. {result.page_start}-{result.page_end}" + ) + source = result.source_path or result.source_url or "source unavailable" + print( + f"[{result.similarity_score:.3f}] {result.source_title} " + f"({pages}; {result.document_type}; synthetic={result.is_synthetic}; source={source})" + ) + if result.section: + print(f"Section: {result.section}") + print(result.text) + print() + return 0 + + +def build_parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser(description="SpecGuard refinery-safety knowledge-base tools") + parser.add_argument("--manifest", type=Path, default=DEFAULT_MANIFEST) + parser.add_argument("--store", type=Path, default=DEFAULT_STORE) + parser.add_argument( + "--embedder", + choices=["deterministic", "sentence-transformers"], + default="deterministic", + ) + subparsers = parser.add_subparsers(dest="command", required=True) + + ingest_parser = subparsers.add_parser("ingest", help="load and index authorized local documents") + ingest_parser.add_argument("--rebuild", action="store_true", help="replace the existing collection") + + subparsers.add_parser("inspect", help="show manifest and vector-store status") + + query_parser = subparsers.add_parser("query", help="retrieve evidence for a natural-language query") + query_parser.add_argument("query_text") + query_parser.add_argument("--top-k", type=int, default=5) + query_parser.add_argument( + "--mode", + choices=["regulations_and_sops", "similar_incidents", "all"], + default="all", + ) + query_parser.add_argument("--risk-type", action="append", choices=[value.value for value in RiskType]) + query_parser.add_argument("--hazard", action="append", choices=[value.value for value in HazardCode]) + query_parser.add_argument("--permit-type", action="append", choices=[value.value for value in PermitType]) + query_parser.add_argument("--document-type", action="append") + return parser + + +def main() -> int: + args = build_parser().parse_args() + if args.command == "ingest": + return ingest(args.manifest, args.store, rebuild=args.rebuild, embedder_name=args.embedder) + if args.command == "inspect": + return inspect(args.manifest, args.store) + return query( + args.query_text, + args.store, + top_k=args.top_k, + mode=args.mode, + embedder_name=args.embedder, + risk_types=args.risk_type, + hazard_codes=args.hazard, + permit_types=args.permit_type, + document_types=args.document_type, + ) + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/backend/app/rag/document_loader.py b/backend/app/rag/document_loader.py new file mode 100644 index 0000000..cf145e3 --- /dev/null +++ b/backend/app/rag/document_loader.py @@ -0,0 +1,108 @@ +"""Load authorized local Markdown, text, and text-based PDF documents.""" + +from __future__ import annotations + +from pathlib import Path + +from app.rag.models import DocumentPage, SourceDocument + + +class DocumentLoadError(RuntimeError): + """Raised when a source cannot be safely loaded.""" + + +def _page(document: SourceDocument, page_number: int, text: str) -> DocumentPage: + return DocumentPage( + document_id=document.document_id, + page_number=page_number, + text=text, + source_title=document.title, + source_path=document.source_path, + source_url=document.source_url, + document_type=document.document_type, + authority=document.authority, + is_synthetic=document.is_synthetic, + publication_date=document.publication_date, + version=document.version, + tags=document.tags, + ) + + +def _resolve_path(document: SourceDocument, base_dir: Path) -> Path: + if not document.source_path: + raise DocumentLoadError( + f"Document {document.document_id} has no local source_path; " + "add an authorized local file before ingesting it." + ) + path = Path(document.source_path) + return path if path.is_absolute() else (base_dir / path) + + +def load_document(document: SourceDocument, base_dir: Path | None = None) -> list[DocumentPage]: + """Extract pages while retaining provenance and skipping empty PDF pages. + + OCR is deliberately not attempted. A scanned PDF with no extractable text + returns an empty list so the ingestion report can flag it for manual review. + """ + + root = (base_dir or Path.cwd()).resolve() + path = _resolve_path(document, root) + if not path.exists(): + raise DocumentLoadError(f"Source file does not exist: {path}") + if not path.is_file(): + raise DocumentLoadError(f"Source path is not a file: {path}") + if path.stat().st_size == 0: + raise DocumentLoadError(f"Source file is empty: {path}") + + suffix = path.suffix.lower() + if suffix in {".md", ".txt"}: + try: + text = path.read_text(encoding="utf-8") + except UnicodeDecodeError as exc: + raise DocumentLoadError(f"Could not decode UTF-8 source file: {path}") from exc + if not text.strip(): + raise DocumentLoadError(f"Source file contains no text: {path}") + return [_page(document, 1, text)] + + if suffix == ".pdf": + try: + from pypdf import PdfReader + except ImportError as exc: + raise DocumentLoadError( + "PDF ingestion requires optional dependency 'pypdf'. " + "Install requirements.txt before loading PDFs." + ) from exc + try: + reader = PdfReader(str(path)) + pages = [ + _page(document, index + 1, page.extract_text() or "") + for index, page in enumerate(reader.pages) + ] + except Exception as exc: # pypdf exposes several parser exception types. + raise DocumentLoadError(f"Could not extract PDF text from {path}: {exc}") from exc + return [page for page in pages if page.text.strip()] + + raise DocumentLoadError( + f"Unsupported document format {suffix or ''!r}; supported formats are .pdf, .md, and .txt" + ) + + +def load_documents( + documents: list[SourceDocument], base_dir: Path | None = None +) -> tuple[list[DocumentPage], list[str]]: + """Load many documents and return pages plus human-readable skip messages.""" + + pages: list[DocumentPage] = [] + skipped: list[str] = [] + for document in documents: + try: + extracted = load_document(document, base_dir=base_dir) + except DocumentLoadError as exc: + skipped.append(str(exc)) + continue + if not extracted: + skipped.append( + f"No extractable text found for {document.document_id}; scanned PDFs need manual OCR review." + ) + pages.extend(extracted) + return pages, skipped diff --git a/backend/app/rag/embedder.py b/backend/app/rag/embedder.py new file mode 100644 index 0000000..eb1c0ba --- /dev/null +++ b/backend/app/rag/embedder.py @@ -0,0 +1,88 @@ +"""Embedding interfaces with an offline deterministic implementation.""" + +from __future__ import annotations + +import hashlib +import math +import os +import re +from abc import ABC, abstractmethod +from collections.abc import Sequence + + +class Embedder(ABC): + """Small interface allowing tests and production models to be swapped.""" + + @property + @abstractmethod + def dimension(self) -> int: + raise NotImplementedError + + @abstractmethod + def embed(self, texts: Sequence[str]) -> list[list[float]]: + raise NotImplementedError + + +class DeterministicEmbedder(Embedder): + """A hash-based local embedder for tests and demos without internet access.""" + + def __init__(self, dimension: int = 128) -> None: + if dimension < 8: + raise ValueError("dimension must be at least 8") + self._dimension = dimension + + @property + def dimension(self) -> int: + return self._dimension + + def embed(self, texts: Sequence[str]) -> list[list[float]]: + vectors: list[list[float]] = [] + for text in texts: + vector = [0.0] * self._dimension + tokens = re.findall(r"[a-z0-9_]+", text.lower()) or [""] + for token in tokens: + digest = hashlib.sha256(token.encode("utf-8")).digest() + index = int.from_bytes(digest[:4], "big") % self._dimension + sign = 1.0 if digest[4] & 1 else -1.0 + vector[index] += sign + norm = math.sqrt(sum(value * value for value in vector)) or 1.0 + vectors.append([value / norm for value in vector]) + return vectors + + +class SentenceTransformerEmbedder(Embedder): + """Optional local Sentence Transformers adapter. + + The model is loaded lazily by the dependency, so importing this module does + not force a large ML install. Downloads are controlled by the user's local + environment and are never required by the offline tests. + """ + + def __init__(self, model_name: str | None = None) -> None: + name = model_name or os.getenv( + "SPECGUARD_EMBEDDING_MODEL", "sentence-transformers/all-MiniLM-L6-v2" + ) + try: + from sentence_transformers import SentenceTransformer + except ImportError as exc: + raise RuntimeError( + "Sentence Transformers is not installed. Use DeterministicEmbedder " + "for offline tests or install the optional ML dependencies." + ) from exc + try: + self._model = SentenceTransformer(name) + except Exception as exc: + raise RuntimeError( + f"Could not load embedding model {name!r}. Check local model access " + "or use the deterministic embedder." + ) from exc + self.model_name = name + self._dimension = int(self._model.get_sentence_embedding_dimension()) + + @property + def dimension(self) -> int: + return self._dimension + + def embed(self, texts: Sequence[str]) -> list[list[float]]: + encoded = self._model.encode(list(texts), normalize_embeddings=True) + return [list(map(float, vector)) for vector in encoded] diff --git a/backend/app/rag/metadata.py b/backend/app/rag/metadata.py new file mode 100644 index 0000000..60dc5c7 --- /dev/null +++ b/backend/app/rag/metadata.py @@ -0,0 +1,109 @@ +"""Manifest parsing and deterministic safety metadata enrichment.""" + +from __future__ import annotations + +import json +from pathlib import Path +from typing import Iterable + +from app.rag.models import DocumentChunk, ManifestEntry +from app.schemas.common import HazardCode, PermitType, RiskType + + +def load_manifest(path: Path) -> list[ManifestEntry]: + """Parse and validate a JSON manifest before ingestion begins.""" + + try: + payload = json.loads(path.read_text(encoding="utf-8")) + except FileNotFoundError as exc: + raise FileNotFoundError(f"Knowledge-base manifest not found: {path}") from exc + if not isinstance(payload, list): + raise ValueError("Knowledge-base manifest must contain a JSON array") + return [ManifestEntry.model_validate(item) for item in payload] + + +_RISK_KEYWORDS: dict[RiskType, tuple[str, ...]] = { + RiskType.FIRE_EXPLOSION: ( + "fire", + "explosion", + "flash fire", + "hydrocarbon", + "ignition", + "hot work", + ), + RiskType.TOXIC_GAS_EXPOSURE: ("h2s", "toxic gas", "poisonous gas"), + RiskType.OXYGEN_DEFICIENCY: ("low oxygen", "oxygen deficiency", "oxygen level", "confined space"), + RiskType.OVERPRESSURE: ("overpressure", "pressure rising", "pressure relief"), + RiskType.EQUIPMENT_FAILURE: ("equipment failure", "pump failure", "malfunction", "ventilation"), +} +_HAZARD_KEYWORDS: dict[HazardCode, tuple[str, ...]] = { + HazardCode.RISING_LEL: ( + "rising lel", + "rising hydrocarbon", + "hydrocarbon concentration", + "hydrocarbon readings", + "gas concentration", + "%lel", + ), + HazardCode.HIGH_H2S: ("h2s", "hydrogen sulfide"), + HazardCode.LOW_OXYGEN: ("low oxygen", "oxygen deficiency"), + HazardCode.PRESSURE_RISING: ("pressure rising", "pressure increase"), + HazardCode.VENTILATION_FAILURE: ( + "ventilation failure", + "ventilation unavailable", + "ventilation is unavailable", + "ventilation failed", + "ventilation has failed", + "ventilation fails", + ), + HazardCode.INCOMPLETE_ISOLATION: ("incomplete isolation", "isolation not verified", "lockout"), + HazardCode.HOT_WORK_ACTIVE: ("hot work", "ignition source", "welding", "cutting"), + HazardCode.CONFINED_SPACE_ACTIVE: ("confined space", "entry permit"), + HazardCode.WORKERS_PRESENT: ( + "workers present", + "workers are present", + "personnel present", + "people are present", + "workers", + "personnel", + ), + HazardCode.OVERDUE_MAINTENANCE: ("overdue maintenance", "maintenance overdue"), +} +_PERMIT_KEYWORDS: dict[PermitType, tuple[str, ...]] = { + PermitType.HOT_WORK: ("hot work permit", "hot work"), + PermitType.CONFINED_SPACE: ("confined space permit", "confined space entry"), + PermitType.LINE_BREAKING: ("line breaking", "line break", "breaking containment"), + PermitType.ELECTRICAL: ("electrical permit", "electrical isolation"), +} +_EQUIPMENT_KEYWORDS = { + "PUMP": ("pump",), + "PIPELINE": ("pipeline", "pipe"), + "VALVE": ("valve",), + "VENTILATION": ("ventilation", "fan"), + "STORAGE_TANK": ("storage tank", "tank"), +} + + +def _matches(text: str, keywords: Iterable[str]) -> bool: + return any(keyword in text for keyword in keywords) + + +def enrich_chunk(chunk: DocumentChunk) -> DocumentChunk: + """Add transparent keyword-derived tags; no model or LLM is involved.""" + + text = f"{chunk.document_title} {chunk.section or ''} {chunk.text}".lower() + risks = [risk for risk, words in _RISK_KEYWORDS.items() if _matches(text, words)] + hazards = [hazard for hazard, words in _HAZARD_KEYWORDS.items() if _matches(text, words)] + permits = [permit for permit, words in _PERMIT_KEYWORDS.items() if _matches(text, words)] + equipment = [name for name, words in _EQUIPMENT_KEYWORDS.items() if _matches(text, words)] + return chunk.model_copy( + update={ + "risk_types": risks, + "hazard_codes": hazards, + "permit_types": permits, + } + ).model_copy(update={"equipment_types": equipment}) + + +def enrich_chunks(chunks: list[DocumentChunk]) -> list[DocumentChunk]: + return [enrich_chunk(chunk) for chunk in chunks] diff --git a/backend/app/rag/models.py b/backend/app/rag/models.py new file mode 100644 index 0000000..11875ca --- /dev/null +++ b/backend/app/rag/models.py @@ -0,0 +1,150 @@ +"""Internal contracts for the knowledge-base pipeline. + +These models describe documents as they move through ingestion and retrieval. +They intentionally stay separate from the public API response models: a chunk +is an implementation detail, while an ``EvidenceReference`` is a frontend/API +contract. +""" + +from datetime import date +from typing import Any + +from pydantic import Field, model_validator + +from app.schemas.common import ( + HazardCode, + NonEmptyString, + PermitType, + RiskType, + SpecGuardSchema, + UsefulText, +) + + +class SourceDocument(SpecGuardSchema): + """Manifest metadata describing one authorized source document.""" + + document_id: NonEmptyString + title: NonEmptyString + document_type: NonEmptyString + authority: NonEmptyString | None = None + source_path: NonEmptyString | None = None + source_url: NonEmptyString | None = None + publication_date: date | None = None + version: NonEmptyString | None = None + is_synthetic: bool = False + industry: NonEmptyString = "PETROLEUM_REFINERY" + tags: list[NonEmptyString] = Field(default_factory=list) + + +class ManifestEntry(SourceDocument): + """A source document plus repository-manifest processing controls.""" + + local_path: NonEmptyString | None = None + allowed_for_public_repo: bool = False + checksum: NonEmptyString | None = None + processing_status: NonEmptyString = "not_added" + + @model_validator(mode="after") + def source_path_matches_local_path(self) -> "ManifestEntry": + if self.source_path is None and self.local_path is not None: + self.source_path = self.local_path + return self + + def to_source_document(self) -> SourceDocument: + """Return only the metadata needed by a document loader.""" + + return SourceDocument( + document_id=self.document_id, + title=self.title, + document_type=self.document_type, + authority=self.authority, + source_path=self.source_path or self.local_path, + source_url=self.source_url, + publication_date=self.publication_date, + version=self.version, + is_synthetic=self.is_synthetic, + industry=self.industry, + tags=self.tags, + ) + + +class DocumentPage(SpecGuardSchema): + """Extracted text for one source page, retaining its provenance.""" + + document_id: NonEmptyString + page_number: int = Field(ge=1) + text: str + source_title: NonEmptyString + source_path: NonEmptyString | None = None + source_url: NonEmptyString | None = None + document_type: NonEmptyString + authority: NonEmptyString | None = None + is_synthetic: bool = False + publication_date: date | None = None + version: NonEmptyString | None = None + tags: list[NonEmptyString] = Field(default_factory=list) + + +class DocumentChunk(SpecGuardSchema): + """A deterministic, retrievable passage with source and safety metadata.""" + + chunk_id: NonEmptyString + document_id: NonEmptyString + document_title: NonEmptyString + text: UsefulText + page_start: int = Field(ge=1) + page_end: int = Field(ge=1) + section: NonEmptyString | None = None + document_type: NonEmptyString + authority: NonEmptyString | None = None + risk_types: list[RiskType] = Field(default_factory=list) + hazard_codes: list[HazardCode] = Field(default_factory=list) + permit_types: list[PermitType] = Field(default_factory=list) + equipment_types: list[NonEmptyString] = Field(default_factory=list) + is_synthetic: bool = False + source_url: NonEmptyString | None = None + source_path: NonEmptyString | None = None + publication_date: date | None = None + version: NonEmptyString | None = None + tags: list[NonEmptyString] = Field(default_factory=list) + + @model_validator(mode="after") + def page_range_is_ordered(self) -> "DocumentChunk": + if self.page_end < self.page_start: + raise ValueError("page_end must be greater than or equal to page_start") + return self + + +class RetrievalQuery(SpecGuardSchema): + """Validated input to the vector store and retriever.""" + + query_text: UsefulText + risk_types: list[RiskType] = Field(default_factory=list) + hazard_codes: list[HazardCode] = Field(default_factory=list) + permit_types: list[PermitType] = Field(default_factory=list) + document_types: list[NonEmptyString] = Field(default_factory=list) + top_k: int = Field(default=5, ge=1, le=20) + + +class RetrievalResult(SpecGuardSchema): + """One ranked chunk returned as evidence for a later safety response.""" + + chunk_id: NonEmptyString + text: UsefulText + similarity_score: float = Field(ge=0.0, le=1.0) + metadata: dict[str, Any] = Field(default_factory=dict) + source_title: NonEmptyString + source_url: NonEmptyString | None = None + source_path: NonEmptyString | None = None + page_start: int = Field(ge=1) + page_end: int = Field(ge=1) + section: NonEmptyString | None = None + document_type: NonEmptyString + is_synthetic: bool = False + + @model_validator(mode="after") + def page_range_is_ordered(self) -> "RetrievalResult": + if self.page_end < self.page_start: + raise ValueError("page_end must be greater than or equal to page_start") + return self diff --git a/backend/app/rag/retriever.py b/backend/app/rag/retriever.py new file mode 100644 index 0000000..8c6bdb0 --- /dev/null +++ b/backend/app/rag/retriever.py @@ -0,0 +1,84 @@ +"""Metadata-aware retrieval and deterministic risk-to-query translation.""" + +from __future__ import annotations + +from app.rag.embedder import Embedder +from app.rag.models import RetrievalQuery, RetrievalResult +from app.rag.vector_store import VectorStore +from app.schemas import RiskEngineInput +from app.schemas.common import HazardCode, PermitType, RiskType, ZoneId + + +_ZONE_NAMES = { + ZoneId.STORAGE_AREA: "storage area", + ZoneId.PUMP_STATION: "pump station", + ZoneId.PIPELINE_AREA: "pipeline area", + ZoneId.MAINTENANCE_AREA: "maintenance area", + ZoneId.CONTROL_ROOM: "control room", +} + + +def risk_to_retrieval_query(risk: RiskEngineInput, top_k: int = 5) -> RetrievalQuery: + """Create a stable evidence query from the existing risk-engine contract.""" + + hazard_words = { + HazardCode.RISING_LEL: "hydrocarbon gas concentration is rising", + HazardCode.HOT_WORK_ACTIVE: "hot work is active", + HazardCode.VENTILATION_FAILURE: "ventilation is unavailable", + HazardCode.WORKERS_PRESENT: "workers are present", + } + details = [hazard_words.get(hazard, hazard.value.replace("_", " ").lower()) for hazard in risk.contributing_factors] + detail_text = ", ".join(details[:-1]) + (f" and {details[-1]}" if len(details) > 1 else (details[0] if details else "")) + query_text = ( + f"{risk.predicted_incident} risk at a refinery {_ZONE_NAMES[risk.zone_id]}. " + f"{detail_text.capitalize()}. Retrieve applicable stop-work precautions, " + "gas-testing requirements, permit suspension, isolation verification, " + "evacuation procedures, reauthorization conditions, and similar incidents." + ) + permits = [PermitType.HOT_WORK] if HazardCode.HOT_WORK_ACTIVE in risk.contributing_factors else [] + return RetrievalQuery( + query_text=query_text, + risk_types=[risk.risk_type], + hazard_codes=list(risk.contributing_factors), + permit_types=permits, + top_k=top_k, + ) + + +class Retriever: + """Embed a query, apply controlled metadata filters, and return evidence.""" + + def __init__(self, store: VectorStore, embedder: Embedder) -> None: + self.store = store + self.embedder = embedder + + def retrieve( + self, + query: RetrievalQuery | str, + *, + mode: str = "all", + top_k: int | None = None, + ) -> list[RetrievalResult]: + if isinstance(query, str): + query = RetrievalQuery(query_text=query, top_k=top_k or 5) + elif top_k is not None: + query = query.model_copy(update={"top_k": top_k}) + if mode not in {"regulations_and_sops", "similar_incidents", "all"}: + raise ValueError("mode must be regulations_and_sops, similar_incidents, or all") + document_types = list(query.document_types) + if mode == "regulations_and_sops": + document_types = ["REGULATION", "SOP"] + elif mode == "similar_incidents": + document_types = ["HISTORICAL_INCIDENT", "NEAR_MISS", "CASE_STUDY"] + elif document_types: + document_types = list(document_types) + filters = { + "risk_types": [value.value for value in query.risk_types], + "hazard_codes": [value.value for value in query.hazard_codes], + "permit_types": [value.value for value in query.permit_types], + "document_type": document_types, + } + vectors = self.embedder.embed([query.query_text]) + if len(vectors) != 1: + raise ValueError("Embedder must return exactly one vector for one query") + return self.store.query(vectors[0], top_k=query.top_k, filters=filters) diff --git a/backend/app/rag/text_cleaner.py b/backend/app/rag/text_cleaner.py new file mode 100644 index 0000000..f6f2940 --- /dev/null +++ b/backend/app/rag/text_cleaner.py @@ -0,0 +1,57 @@ +"""Conservative safety-text normalization.""" + +from __future__ import annotations + +import re +from collections import Counter + +from app.rag.models import DocumentPage + + +def clean_text(text: str) -> str: + """Normalize layout noise without paraphrasing safety instructions.""" + + # Null bytes and CRLF are extraction artifacts; preserving words and + # punctuation matters more than making prose look cosmetically perfect. + text = text.replace("\x00", "").replace("\r\n", "\n").replace("\r", "\n") + # Only join an obvious word split at a line boundary. We do not remove + # arbitrary hyphens because equipment identifiers and units may contain them. + text = re.sub(r"(?<=\w)-\n(?=\w)", "", text) + lines = [re.sub(r"[ \t]+", " ", line).strip() for line in text.split("\n")] + output: list[str] = [] + previous_blank = False + for line in lines: + if not line: + if not previous_blank: + output.append("") + previous_blank = True + continue + output.append(line) + previous_blank = False + return "\n".join(output).strip() + + +def clean_pages(pages: list[DocumentPage]) -> list[DocumentPage]: + """Clean pages and remove only headers/footers repeated on multiple pages.""" + + cleaned = [page.model_copy(update={"text": clean_text(page.text)}) for page in pages] + if len(cleaned) < 2: + return cleaned + + first_lines = [page.text.splitlines()[0] for page in cleaned if page.text.splitlines()] + last_lines = [page.text.splitlines()[-1] for page in cleaned if page.text.splitlines()] + repeated = { + line for line, count in Counter(first_lines).items() if count >= 2 + } | {line for line, count in Counter(last_lines).items() if count >= 2} + if not repeated: + return cleaned + + result: list[DocumentPage] = [] + for page in cleaned: + lines = page.text.splitlines() + if lines and lines[0] in repeated: + lines = lines[1:] + if lines and lines[-1] in repeated: + lines = lines[:-1] + result.append(page.model_copy(update={"text": clean_text("\n".join(lines))})) + return result diff --git a/backend/app/rag/vector_store.py b/backend/app/rag/vector_store.py new file mode 100644 index 0000000..c7c71bc --- /dev/null +++ b/backend/app/rag/vector_store.py @@ -0,0 +1,179 @@ +"""Persistent vector-store abstraction with a dependency-free JSON backend.""" + +from __future__ import annotations + +import json +import math +from abc import ABC, abstractmethod +from pathlib import Path +from typing import Any + +from app.rag.models import DocumentChunk, RetrievalResult + + +def _cosine(left: list[float], right: list[float]) -> float: + if len(left) != len(right): + raise ValueError("Embedding dimensions do not match") + numerator = sum(a * b for a, b in zip(left, right)) + left_norm = math.sqrt(sum(value * value for value in left)) + right_norm = math.sqrt(sum(value * value for value in right)) + if not left_norm or not right_norm: + return 0.0 + # Cosine may be negative; retrieval scores are exposed as a relevance range. + return max(0.0, min(1.0, numerator / (left_norm * right_norm))) + + +class VectorStore(ABC): + """Backend-neutral operations needed by the retriever.""" + + @abstractmethod + def add(self, chunks: list[DocumentChunk], embeddings: list[list[float]]) -> None: + raise NotImplementedError + + @abstractmethod + def delete_documents(self, document_ids: list[str]) -> None: + raise NotImplementedError + + @abstractmethod + def query( + self, + embedding: list[float], + *, + top_k: int, + filters: dict[str, list[str]] | None = None, + ) -> list[RetrievalResult]: + raise NotImplementedError + + +class JsonVectorStore(VectorStore): + """Small persistent store used when Chroma is not part of the project. + + The file format is intentionally inspectable. A future Chroma adapter can + implement the same ``VectorStore`` interface without changing the retriever. + """ + + def __init__(self, path: Path) -> None: + self.path = path + self._records: list[dict[str, Any]] = [] + self._embedding_dimension: int | None = None + self._load() + + def _load(self) -> None: + if not self.path.exists(): + return + payload = json.loads(self.path.read_text(encoding="utf-8")) + if not isinstance(payload, list): + raise ValueError(f"Vector store must contain a JSON array: {self.path}") + self._records = payload + dimensions = {len(record.get("embedding", [])) for record in self._records} + if len(dimensions) > 1 or (dimensions and 0 in dimensions): + raise ValueError(f"Vector store contains inconsistent or empty embeddings: {self.path}") + self._embedding_dimension = next(iter(dimensions), None) + + def _persist(self) -> None: + self.path.parent.mkdir(parents=True, exist_ok=True) + self.path.write_text(json.dumps(self._records, indent=2), encoding="utf-8") + + @property + def count(self) -> int: + return len(self._records) + + @property + def embedding_dimension(self) -> int | None: + return self._embedding_dimension + + def add(self, chunks: list[DocumentChunk], embeddings: list[list[float]]) -> None: + if len(chunks) != len(embeddings): + raise ValueError("Each chunk must have exactly one embedding") + if not chunks: + return + dimensions = {len(embedding) for embedding in embeddings} + if not dimensions or 0 in dimensions or len(dimensions) != 1: + raise ValueError("All embeddings must be non-empty and have one shared dimension") + dimension = next(iter(dimensions)) + if self._embedding_dimension is not None and dimension != self._embedding_dimension: + raise ValueError( + f"Embedding dimension {dimension} does not match store dimension " + f"{self._embedding_dimension}" + ) + by_id = {record["chunk"]["chunk_id"]: record for record in self._records} + for chunk, embedding in zip(chunks, embeddings): + if not embedding: + raise ValueError(f"Embedding is empty for chunk {chunk.chunk_id}") + by_id[chunk.chunk_id] = { + "chunk": chunk.model_dump(mode="json"), + "embedding": [float(value) for value in embedding], + } + self._records = list(by_id.values()) + self._embedding_dimension = dimension + self._persist() + + def rebuild(self, chunks: list[DocumentChunk], embeddings: list[list[float]]) -> None: + self._records = [] + self._embedding_dimension = None + if not chunks: + self._persist() + return + self.add(chunks, embeddings) + + def delete_documents(self, document_ids: list[str]) -> None: + targets = set(document_ids) + self._records = [ + record for record in self._records if record["chunk"].get("document_id") not in targets + ] + self._persist() + + @staticmethod + def _matches(chunk: DocumentChunk, filters: dict[str, list[str]]) -> bool: + for key, wanted in filters.items(): + if not wanted: + continue + if key == "document_type": + if chunk.document_type.upper() not in {value.upper() for value in wanted}: + return False + continue + values = { + str(value.value if hasattr(value, "value") else value).upper() + for value in getattr(chunk, key, []) + } + # A risk query commonly contains several contributing hazards. A + # chunk matching any requested value is useful evidence; requiring + # every tag would hide focused SOP sections that cover one control. + if not values.intersection({value.upper() for value in wanted}): + return False + return True + + def query( + self, + embedding: list[float], + *, + top_k: int, + filters: dict[str, list[str]] | None = None, + ) -> list[RetrievalResult]: + if top_k < 1: + raise ValueError("top_k must be positive") + results: list[RetrievalResult] = [] + for record in self._records: + chunk = DocumentChunk.model_validate(record["chunk"]) + if filters and not self._matches(chunk, filters): + continue + score = _cosine(embedding, [float(value) for value in record["embedding"]]) + metadata = chunk.model_dump(mode="json") + results.append( + RetrievalResult( + chunk_id=chunk.chunk_id, + text=chunk.text, + similarity_score=score, + metadata=metadata, + source_title=chunk.document_title, + source_url=chunk.source_url, + source_path=chunk.source_path, + page_start=chunk.page_start, + page_end=chunk.page_end, + section=chunk.section, + document_type=chunk.document_type, + is_synthetic=chunk.is_synthetic, + ) + ) + results.sort(key=lambda result: (-result.similarity_score, result.chunk_id)) + return results[:top_k] diff --git a/backend/app/schemas/__init__.py b/backend/app/schemas/__init__.py index e69de29..8884bda 100644 --- a/backend/app/schemas/__init__.py +++ b/backend/app/schemas/__init__.py @@ -0,0 +1,48 @@ +"""Public Pydantic contracts for the SpecGuard backend and services.""" + +from app.schemas.common import ( + ActionStatus, + EvidenceType, + HazardCode, + MaintenanceStatus, + PermitStatus, + PermitType, + RiskType, + Severity, + ZoneId, +) +from app.schemas.incident import ( + HistoricalIncident, + IncidentSearchQuery, + NearMissRecord, + SimilarIncident, +) +from app.schemas.intelligence import ( + EvidenceReference, + RecommendedAction, + RiskEngineInput, + SafetyIntelligenceResponse, +) +from app.schemas.shift_log import ExtractedSafetyEvent, ShiftLogEntry + +__all__ = [ + "ActionStatus", + "EvidenceReference", + "EvidenceType", + "ExtractedSafetyEvent", + "HazardCode", + "HistoricalIncident", + "IncidentSearchQuery", + "MaintenanceStatus", + "NearMissRecord", + "PermitStatus", + "PermitType", + "RecommendedAction", + "RiskEngineInput", + "RiskType", + "SafetyIntelligenceResponse", + "Severity", + "ShiftLogEntry", + "SimilarIncident", + "ZoneId", +] diff --git a/backend/app/schemas/common.py b/backend/app/schemas/common.py new file mode 100644 index 0000000..c0d0745 --- /dev/null +++ b/backend/app/schemas/common.py @@ -0,0 +1,126 @@ +"""Shared vocabulary and validation behavior for SpecGuard contracts.""" + +from enum import Enum +from typing import Annotated + +from pydantic import BaseModel, ConfigDict, StringConstraints + + +# A controlled enum prevents different services from representing the same +# concept as "hot-work", "Hot Work", and "HOT_WORK". Stable values make joins, +# filtering, alert rules, and frontend rendering predictable across the system. +class Severity(str, Enum): + LOW = "LOW" + MEDIUM = "MEDIUM" + HIGH = "HIGH" + CRITICAL = "CRITICAL" + + +class ZoneId(str, Enum): + """Prototype refinery zones with stable serialized identifiers.""" + + STORAGE_AREA = "ZONE_A" + PUMP_STATION = "ZONE_B" + PIPELINE_AREA = "ZONE_C" + MAINTENANCE_AREA = "ZONE_D" + CONTROL_ROOM = "ZONE_E" + + +class RiskType(str, Enum): + FIRE_EXPLOSION = "FIRE_EXPLOSION" + TOXIC_GAS_EXPOSURE = "TOXIC_GAS_EXPOSURE" + OXYGEN_DEFICIENCY = "OXYGEN_DEFICIENCY" + OVERPRESSURE = "OVERPRESSURE" + EQUIPMENT_FAILURE = "EQUIPMENT_FAILURE" + CONFINED_SPACE = "CONFINED_SPACE" + ELECTRICAL_HAZARD = "ELECTRICAL_HAZARD" + UNKNOWN = "UNKNOWN" + + +class HazardCode(str, Enum): + RISING_LEL = "RISING_LEL" + HIGH_H2S = "HIGH_H2S" + LOW_OXYGEN = "LOW_OXYGEN" + PRESSURE_RISING = "PRESSURE_RISING" + HIGH_TEMPERATURE = "HIGH_TEMPERATURE" + ABNORMAL_FLOW = "ABNORMAL_FLOW" + VENTILATION_FAILURE = "VENTILATION_FAILURE" + INCOMPLETE_ISOLATION = "INCOMPLETE_ISOLATION" + HOT_WORK_ACTIVE = "HOT_WORK_ACTIVE" + CONFINED_SPACE_ACTIVE = "CONFINED_SPACE_ACTIVE" + WORKERS_PRESENT = "WORKERS_PRESENT" + PPE_VIOLATION = "PPE_VIOLATION" + RESTRICTED_ZONE_ENTRY = "RESTRICTED_ZONE_ENTRY" + OVERDUE_MAINTENANCE = "OVERDUE_MAINTENANCE" + UNRESOLVED_SHIFT_OBSERVATION = "UNRESOLVED_SHIFT_OBSERVATION" + UNKNOWN = "UNKNOWN" + + +class PermitType(str, Enum): + HOT_WORK = "HOT_WORK" + COLD_WORK = "COLD_WORK" + CONFINED_SPACE = "CONFINED_SPACE" + LINE_BREAKING = "LINE_BREAKING" + ELECTRICAL = "ELECTRICAL" + WORKING_AT_HEIGHT = "WORKING_AT_HEIGHT" + + +class PermitStatus(str, Enum): + DRAFT = "DRAFT" + REQUESTED = "REQUESTED" + APPROVED = "APPROVED" + ACTIVE = "ACTIVE" + SUSPENDED = "SUSPENDED" + EXPIRED = "EXPIRED" + CLOSED = "CLOSED" + CANCELLED = "CANCELLED" + + +class MaintenanceStatus(str, Enum): + SCHEDULED = "SCHEDULED" + IN_PROGRESS = "IN_PROGRESS" + COMPLETED = "COMPLETED" + OVERDUE = "OVERDUE" + CANCELLED = "CANCELLED" + + +class ActionStatus(str, Enum): + PROPOSED = "PROPOSED" + APPROVED = "APPROVED" + REJECTED = "REJECTED" + IN_PROGRESS = "IN_PROGRESS" + COMPLETED = "COMPLETED" + FAILED = "FAILED" + + +class EvidenceType(str, Enum): + REGULATION = "REGULATION" + SOP = "SOP" + HISTORICAL_INCIDENT = "HISTORICAL_INCIDENT" + NEAR_MISS = "NEAR_MISS" + SHIFT_LOG = "SHIFT_LOG" + SENSOR = "SENSOR" + PERMIT = "PERMIT" + MAINTENANCE = "MAINTENANCE" + CCTV = "CCTV" + + +NonEmptyString = Annotated[ + str, + StringConstraints(strip_whitespace=True, min_length=1), +] +UsefulText = Annotated[ + str, + StringConstraints(strip_whitespace=True, min_length=10), +] + + +class SpecGuardSchema(BaseModel): + """Base behavior shared by the new public service contracts. + + Rejecting unknown fields catches producer/consumer version drift instead of + silently discarding safety-relevant data. Whitespace stripping also makes a + string containing only spaces fail the normal minimum-length checks. + """ + + model_config = ConfigDict(extra="forbid", str_strip_whitespace=True) diff --git a/backend/app/schemas/incident.py b/backend/app/schemas/incident.py new file mode 100644 index 0000000..e01844b --- /dev/null +++ b/backend/app/schemas/incident.py @@ -0,0 +1,95 @@ +"""Historical incident, near-miss, and similarity-search contracts.""" + +from datetime import datetime + +from pydantic import Field + +from app.schemas.common import ( + HazardCode, + NonEmptyString, + PermitType, + RiskType, + Severity, + SpecGuardSchema, + UsefulText, + ZoneId, +) + + +class HistoricalIncident(SpecGuardSchema): + """A normalized incident used for retrieval and prevention analysis.""" + + incident_id: NonEmptyString + title: NonEmptyString + occurred_at: datetime | None = None + industry: NonEmptyString = "PETROLEUM_REFINERY" + facility_type: NonEmptyString = "REFINERY_PROCESS_UNIT" + zone_id: ZoneId | None = None + equipment_ids: list[NonEmptyString] = Field(default_factory=list) + risk_type: RiskType + severity: Severity + permit_types: list[PermitType] = Field(default_factory=list) + contributing_hazards: list[HazardCode] = Field(default_factory=list) + summary: UsefulText + root_causes: list[NonEmptyString] = Field(default_factory=list) + consequences: list[NonEmptyString] = Field(default_factory=list) + preventive_actions: list[NonEmptyString] = Field(default_factory=list) + source_title: NonEmptyString | None = None + source_url: NonEmptyString | None = None + source_page: int | None = Field(default=None, ge=1) + is_synthetic: bool + + +class NearMissRecord(SpecGuardSchema): + """A hazardous event that did not result in an incident. + + Near misses are modeled separately because their potential severity differs + from their actual outcome. They provide valuable weak signals for prevention + even when nobody was injured and no equipment was damaged. + """ + + near_miss_id: NonEmptyString + title: NonEmptyString + occurred_at: datetime | None = None + zone_id: ZoneId | None = None + equipment_ids: list[NonEmptyString] = Field(default_factory=list) + risk_type: RiskType + potential_severity: Severity + permit_types: list[PermitType] = Field(default_factory=list) + contributing_hazards: list[HazardCode] = Field(default_factory=list) + description: UsefulText + immediate_actions: list[NonEmptyString] = Field(default_factory=list) + recommended_preventive_actions: list[NonEmptyString] = Field( + default_factory=list + ) + source_title: NonEmptyString | None = None + source_url: NonEmptyString | None = None + source_page: int | None = Field(default=None, ge=1) + is_synthetic: bool + + +class IncidentSearchQuery(SpecGuardSchema): + """Filters and free text supplied to incident-similarity retrieval.""" + + risk_type: RiskType | None = None + zone_id: ZoneId | None = None + equipment_ids: list[NonEmptyString] = Field(default_factory=list) + hazards: list[HazardCode] = Field(default_factory=list) + active_permits: list[PermitType] = Field(default_factory=list) + natural_language_query: NonEmptyString | None = None + top_k: int = Field(default=5, ge=1, le=20) + + +class SimilarIncident(SpecGuardSchema): + """A compact historical result returned by the retrieval pipeline.""" + + incident_id: NonEmptyString + title: NonEmptyString + similarity_score: float = Field(ge=0.0, le=1.0) + shared_hazards: list[HazardCode] = Field(default_factory=list) + summary: UsefulText + root_causes: list[NonEmptyString] = Field(default_factory=list) + preventive_actions: list[NonEmptyString] = Field(default_factory=list) + source_title: NonEmptyString | None = None + source_url: NonEmptyString | None = None + source_page: int | None = Field(default=None, ge=1) diff --git a/backend/app/schemas/intelligence.py b/backend/app/schemas/intelligence.py new file mode 100644 index 0000000..2b3f7bd --- /dev/null +++ b/backend/app/schemas/intelligence.py @@ -0,0 +1,99 @@ +"""Contracts joining risk detection, evidence retrieval, and recommendations.""" + +from datetime import datetime +from typing import Any + +from pydantic import Field + +from app.schemas.common import ( + ActionStatus, + EvidenceType, + HazardCode, + NonEmptyString, + RiskType, + Severity, + SpecGuardSchema, + UsefulText, + ZoneId, +) +from app.schemas.incident import SimilarIncident + + +class RiskEngineInput(SpecGuardSchema): + """The stable handoff from risk detection to safety intelligence. + + Rex's compound-risk engine produces this object. Ashish's NLP/RAG pipeline + consumes it. The explicit contract prevents hidden coupling between those + independently developed components. + """ + + alert_id: NonEmptyString + timestamp: datetime + zone_id: ZoneId + equipment_ids: list[NonEmptyString] = Field(min_length=1) + risk_type: RiskType + risk_score: float = Field(ge=0.0, le=1.0) + severity: Severity + predicted_incident: UsefulText + contributing_factors: list[HazardCode] = Field(min_length=1) + # Evidence keys vary by sensor family, so this boundary deliberately stays + # flexible rather than pretending every process variable is one sensor type. + sensor_evidence: dict[str, Any] = Field(min_length=1) + active_permit_ids: list[NonEmptyString] = Field(default_factory=list) + maintenance_event_ids: list[NonEmptyString] = Field(default_factory=list) + shift_log_ids: list[NonEmptyString] = Field(default_factory=list) + cctv_event_ids: list[NonEmptyString] = Field(default_factory=list) + estimated_lead_time_minutes: float | None = Field(default=None, ge=0.0) + model_confidence: float | None = Field(default=None, ge=0.0, le=1.0) + + +class EvidenceReference(SpecGuardSchema): + """A compact pointer to source material used by the intelligence pipeline. + + Evidence IDs let actions cite relevant material without copying entire SOPs, + regulations, sensor payloads, or incident documents into every response. + """ + + evidence_id: NonEmptyString + evidence_type: EvidenceType + title: NonEmptyString + excerpt: UsefulText + source_name: NonEmptyString + source_url: NonEmptyString | None = None + page_number: int | None = Field(default=None, ge=1) + section_name: NonEmptyString | None = None + relevance_score: float = Field(ge=0.0, le=1.0) + + +class RecommendedAction(SpecGuardSchema): + """An advisory response step that remains under human control. + + SpecGuard does not model an LLM as directly controlling refinery equipment. + A qualified human must review proposed actions before operational execution. + """ + + action_id: NonEmptyString + title: NonEmptyString + description: UsefulText + priority: int = Field(ge=1, le=10) + status: ActionStatus + requires_human_approval: bool = True + target_role: NonEmptyString + supporting_evidence_ids: list[NonEmptyString] = Field(default_factory=list) + + +class SafetyIntelligenceResponse(SpecGuardSchema): + """Final safety-intelligence response returned to backend and frontend.""" + + intelligence_id: NonEmptyString + alert_id: NonEmptyString + generated_at: datetime + executive_summary: UsefulText + risk_explanation: UsefulText + recommended_actions: list[RecommendedAction] = Field(default_factory=list) + evidence: list[EvidenceReference] = Field(default_factory=list) + similar_incidents: list[SimilarIncident] = Field(default_factory=list) + intelligence_confidence: float = Field(ge=0.0, le=1.0) + insufficient_evidence: bool + limitations: list[NonEmptyString] = Field(default_factory=list) + requires_human_review: bool = True diff --git a/backend/app/schemas/shift_log.py b/backend/app/schemas/shift_log.py new file mode 100644 index 0000000..276dcf5 --- /dev/null +++ b/backend/app/schemas/shift_log.py @@ -0,0 +1,50 @@ +"""Contracts for raw operator logs and NLP-derived safety information.""" + +from datetime import datetime + +from pydantic import Field + +from app.schemas.common import ( + HazardCode, + NonEmptyString, + Severity, + SpecGuardSchema, + UsefulText, + ZoneId, +) + + +class ShiftLogEntry(SpecGuardSchema): + """An operator's original shift observation before NLP processing. + + Raw text is retained separately from extracted facts so that reviewers can + audit the source statement and rerun improved extraction logic later. + """ + + log_id: NonEmptyString + timestamp: datetime + shift_id: NonEmptyString + author_role: NonEmptyString + zone_id: ZoneId | None = None + equipment_ids: list[NonEmptyString] = Field(default_factory=list) + raw_text: UsefulText + acknowledged: bool + resolved: bool + + +class ExtractedSafetyEvent(SpecGuardSchema): + """Structured safety facts produced by NLP from one raw shift log. + + The extracted event is a machine-friendly interpretation, not a replacement + for the source log. Its confidence and method expose extraction uncertainty. + """ + + source_log_id: NonEmptyString + zone_id: ZoneId | None = None + equipment_ids: list[NonEmptyString] = Field(default_factory=list) + hazards: list[HazardCode] = Field(default_factory=list) + severity: Severity + confidence: float = Field(ge=0.0, le=1.0) + summary: UsefulText + requires_follow_up: bool + extraction_method: NonEmptyString = "RULE_LLM_HYBRID" diff --git a/backend/data/knowledge/manifests/documents.json b/backend/data/knowledge/manifests/documents.json new file mode 100644 index 0000000..8b5ff69 --- /dev/null +++ b/backend/data/knowledge/manifests/documents.json @@ -0,0 +1,272 @@ +[ + { + "document_id": "SYN-SOP-HOT-WORK", + "title": "Synthetic Hot Work Safety Procedure", + "document_type": "SOP", + "authority": "SpecGuard Hackathon Prototype", + "source_url": null, + "local_path": "raw/synthetic_sops/hot_work_sop.md", + "publication_date": null, + "version": "prototype-1", + "is_synthetic": true, + "allowed_for_public_repo": true, + "checksum": null, + "tags": ["hot work", "gas testing", "fire explosion"], + "processing_status": "ready" + }, + { + "document_id": "SYN-SOP-GAS-TESTING", + "title": "Synthetic Gas Testing and Atmosphere Monitoring Procedure", + "document_type": "SOP", + "authority": "SpecGuard Hackathon Prototype", + "source_url": null, + "local_path": "raw/synthetic_sops/gas_testing_sop.md", + "publication_date": null, + "version": "prototype-1", + "is_synthetic": true, + "allowed_for_public_repo": true, + "checksum": null, + "tags": ["gas testing", "rising lel", "h2s", "oxygen"], + "processing_status": "ready" + }, + { + "document_id": "SYN-SOP-EQUIPMENT-ISOLATION", + "title": "Synthetic Equipment Isolation Procedure", + "document_type": "SOP", + "authority": "SpecGuard Hackathon Prototype", + "source_url": null, + "local_path": "raw/synthetic_sops/equipment_isolation_sop.md", + "publication_date": null, + "version": "prototype-1", + "is_synthetic": true, + "allowed_for_public_repo": true, + "checksum": null, + "tags": ["isolation", "lockout", "line breaking"], + "processing_status": "ready" + }, + { + "document_id": "SYN-SOP-LINE-BREAKING", + "title": "Synthetic Line Breaking Procedure", + "document_type": "SOP", + "authority": "SpecGuard Hackathon Prototype", + "source_url": null, + "local_path": "raw/synthetic_sops/line_breaking_sop.md", + "publication_date": null, + "version": "prototype-1", + "is_synthetic": true, + "allowed_for_public_repo": true, + "checksum": null, + "tags": ["line breaking", "containment", "toxic gas"], + "processing_status": "ready" + }, + { + "document_id": "SYN-SOP-VENTILATION-FAILURE", + "title": "Synthetic Ventilation Failure Response Procedure", + "document_type": "SOP", + "authority": "SpecGuard Hackathon Prototype", + "source_url": null, + "local_path": "raw/synthetic_sops/ventilation_failure_sop.md", + "publication_date": null, + "version": "prototype-1", + "is_synthetic": true, + "allowed_for_public_repo": true, + "checksum": null, + "tags": ["ventilation failure", "workers present", "stop work"], + "processing_status": "ready" + }, + { + "document_id": "SYN-SOP-EMERGENCY-EVACUATION", + "title": "Synthetic Emergency Evacuation Procedure", + "document_type": "SOP", + "authority": "SpecGuard Hackathon Prototype", + "source_url": null, + "local_path": "raw/synthetic_sops/emergency_evacuation_sop.md", + "publication_date": null, + "version": "prototype-1", + "is_synthetic": true, + "allowed_for_public_repo": true, + "checksum": null, + "tags": ["evacuation", "fire explosion", "toxic gas"], + "processing_status": "ready" + }, + { + "document_id": "LOCAL-CASE-EXPLOSION-IN-FURNACE", + "title": "Explosion in furnace (local copy)", + "document_type": "HISTORICAL_INCIDENT", + "authority": "Local supplied source; authority pending verification", + "source_url": null, + "local_path": "raw/case_studies/Explosion in furnace.pdf", + "publication_date": null, + "version": null, + "is_synthetic": false, + "allowed_for_public_repo": false, + "checksum": null, + "tags": ["furnace", "explosion", "case study"], + "processing_status": "pending_review" + }, + { + "document_id": "LOCAL-CASE-OISD-ADSORBENT-REMOVAL-FATALITY", + "title": "OISD adsorbent removal fatality (local copy)", + "document_type": "HISTORICAL_INCIDENT", + "authority": "Local supplied source; authority pending verification", + "source_url": null, + "local_path": "raw/case_studies/oisd_adsorbent_removal_fatality.pdf", + "publication_date": null, + "version": null, + "is_synthetic": false, + "allowed_for_public_repo": false, + "checksum": null, + "tags": ["fatality", "maintenance", "case study"], + "processing_status": "pending_review" + }, + { + "document_id": "LOCAL-CASE-OISD-BENZENE-STORAGE-TANK-EXPLOSION", + "title": "OISD benzene storage tank explosion (local copy)", + "document_type": "HISTORICAL_INCIDENT", + "authority": "Local supplied source; authority pending verification", + "source_url": null, + "local_path": "raw/case_studies/oisd_benzene_storage_tank_explosion.pdf", + "publication_date": null, + "version": null, + "is_synthetic": false, + "allowed_for_public_repo": false, + "checksum": null, + "tags": ["benzene", "storage tank", "explosion"], + "processing_status": "pending_review" + }, + { + "document_id": "LOCAL-CASE-OISD-ETHANOL-TANK-EXPLOSION", + "title": "OISD ethanol tank explosion (local copy)", + "document_type": "HISTORICAL_INCIDENT", + "authority": "Local supplied source; authority pending verification", + "source_url": null, + "local_path": "raw/case_studies/oisd_ethanol_tank_explosion.pdf", + "publication_date": null, + "version": null, + "is_synthetic": false, + "allowed_for_public_repo": false, + "checksum": null, + "tags": ["ethanol", "tank", "explosion"], + "processing_status": "pending_review" + }, + { + "document_id": "LOCAL-CASE-OISD-FIRE-ATMOSPHERIC-VACUUM-UNIT", + "title": "OISD fire in atmospheric vacuum unit (local copy)", + "document_type": "HISTORICAL_INCIDENT", + "authority": "Local supplied source; authority pending verification", + "source_url": null, + "local_path": "raw/case_studies/oisd_fire_atmospheric_vacuum_unit.pdf", + "publication_date": null, + "version": null, + "is_synthetic": false, + "allowed_for_public_repo": false, + "checksum": null, + "tags": ["fire", "vacuum unit", "case study"], + "processing_status": "pending_review" + }, + { + "document_id": "LOCAL-CASE-OISD-FIRE-COMPRESSOR-HOUSE", + "title": "OISD fire in compressor house (local copy)", + "document_type": "HISTORICAL_INCIDENT", + "authority": "Local supplied source; authority pending verification", + "source_url": null, + "local_path": "raw/case_studies/oisd_fire_compressor_house.pdf", + "publication_date": null, + "version": null, + "is_synthetic": false, + "allowed_for_public_repo": false, + "checksum": null, + "tags": ["fire", "compressor", "case study"], + "processing_status": "pending_review" + }, + { + "document_id": "LOCAL-CASE-OISD-H2S-FATALITY-FLOATING-ROOF-TANK", + "title": "OISD H2S fatality at floating roof tank (local copy)", + "document_type": "HISTORICAL_INCIDENT", + "authority": "Local supplied source; authority pending verification", + "source_url": null, + "local_path": "raw/case_studies/oisd_h2s_fatality_floating_roof_tank.pdf", + "publication_date": null, + "version": null, + "is_synthetic": false, + "allowed_for_public_repo": false, + "checksum": null, + "tags": ["h2s", "fatality", "storage tank"], + "processing_status": "pending_review" + }, + { + "document_id": "LOCAL-CASE-OISD-HYDROCARBON-DRAIN-LINE-FIRE", + "title": "OISD hydrocarbon drain line fire (local copy)", + "document_type": "HISTORICAL_INCIDENT", + "authority": "Local supplied source; authority pending verification", + "source_url": null, + "local_path": "raw/case_studies/oisd_hydrocarbon_drain_line_fire.pdf", + "publication_date": null, + "version": null, + "is_synthetic": false, + "allowed_for_public_repo": false, + "checksum": null, + "tags": ["hydrocarbon", "drain line", "fire"], + "processing_status": "pending_review" + }, + { + "document_id": "LOCAL-CASE-OISD-NAPHTHA-REBOILER-FURNACE-FIRE", + "title": "OISD naphtha reboiler furnace fire (local copy)", + "document_type": "HISTORICAL_INCIDENT", + "authority": "Local supplied source; authority pending verification", + "source_url": null, + "local_path": "raw/case_studies/oisd_naphtha_reboiler_furnace_fire.pdf", + "publication_date": null, + "version": null, + "is_synthetic": false, + "allowed_for_public_repo": false, + "checksum": null, + "tags": ["naphtha", "furnace", "fire"], + "processing_status": "pending_review" + }, + { + "document_id": "LOCAL-CASE-OISD-WET-SLOP-PUMP-HOUSE-FIRE", + "title": "OISD wet slop pump house fire (local copy)", + "document_type": "HISTORICAL_INCIDENT", + "authority": "Local supplied source; authority pending verification", + "source_url": null, + "local_path": "raw/case_studies/oisd_wet_slop_pump_house_fire.pdf", + "publication_date": null, + "version": null, + "is_synthetic": false, + "allowed_for_public_repo": false, + "checksum": null, + "tags": ["pump house", "slop", "fire"], + "processing_status": "pending_review" + }, + { + "document_id": "LOCAL-REG-OISD-STD-116-FIRE-PROTECTION", + "title": "OISD Standard 116 fire protection (local copy)", + "document_type": "REGULATION", + "authority": "Local supplied source; authority pending verification", + "source_url": null, + "local_path": "raw/regulations/oisd_std_116_fire_protection.pdf", + "publication_date": null, + "version": null, + "is_synthetic": false, + "allowed_for_public_repo": false, + "checksum": null, + "tags": ["fire protection", "standard", "regulation"], + "processing_status": "pending_review" + }, + { + "document_id": "LOCAL-REG-PNGRB-HALDIA-REFINERY-NAPHTHA-RELEASE", + "title": "PNGRB Haldia refinery naphtha release (local copy)", + "document_type": "REGULATION", + "authority": "Local supplied source; authority pending verification", + "source_url": null, + "local_path": "raw/regulations/pngrb_haldia_refinery_naphtha_release.pdf", + "publication_date": null, + "version": null, + "is_synthetic": false, + "allowed_for_public_repo": false, + "checksum": null, + "tags": ["naphtha", "release", "regulation"], + "processing_status": "pending_review" + } +] diff --git a/backend/data/knowledge/processed/.gitkeep b/backend/data/knowledge/processed/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/backend/data/knowledge/raw/case_studies/.gitkeep b/backend/data/knowledge/raw/case_studies/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/backend/data/knowledge/raw/case_studies/oisd_adsorbent_removal_fatality.pdf b/backend/data/knowledge/raw/case_studies/oisd_adsorbent_removal_fatality.pdf new file mode 100644 index 0000000..7de8b3b Binary files /dev/null and b/backend/data/knowledge/raw/case_studies/oisd_adsorbent_removal_fatality.pdf differ diff --git a/backend/data/knowledge/raw/case_studies/oisd_benzene_storage_tank_explosion.pdf b/backend/data/knowledge/raw/case_studies/oisd_benzene_storage_tank_explosion.pdf new file mode 100644 index 0000000..3c98c9f Binary files /dev/null and b/backend/data/knowledge/raw/case_studies/oisd_benzene_storage_tank_explosion.pdf differ diff --git a/backend/data/knowledge/raw/case_studies/oisd_ethanol_tank_explosion.pdf b/backend/data/knowledge/raw/case_studies/oisd_ethanol_tank_explosion.pdf new file mode 100644 index 0000000..318da36 Binary files /dev/null and b/backend/data/knowledge/raw/case_studies/oisd_ethanol_tank_explosion.pdf differ diff --git a/backend/data/knowledge/raw/case_studies/oisd_explosion_in_furnace.pdf b/backend/data/knowledge/raw/case_studies/oisd_explosion_in_furnace.pdf new file mode 100644 index 0000000..7256920 Binary files /dev/null and b/backend/data/knowledge/raw/case_studies/oisd_explosion_in_furnace.pdf differ diff --git a/backend/data/knowledge/raw/case_studies/oisd_fire_atmospheric_vacuum_unit.pdf b/backend/data/knowledge/raw/case_studies/oisd_fire_atmospheric_vacuum_unit.pdf new file mode 100644 index 0000000..6ee162b Binary files /dev/null and b/backend/data/knowledge/raw/case_studies/oisd_fire_atmospheric_vacuum_unit.pdf differ diff --git a/backend/data/knowledge/raw/case_studies/oisd_fire_compressor_house.pdf b/backend/data/knowledge/raw/case_studies/oisd_fire_compressor_house.pdf new file mode 100644 index 0000000..1848c7b Binary files /dev/null and b/backend/data/knowledge/raw/case_studies/oisd_fire_compressor_house.pdf differ diff --git a/backend/data/knowledge/raw/case_studies/oisd_h2s_fatality_floating_roof_tank.pdf b/backend/data/knowledge/raw/case_studies/oisd_h2s_fatality_floating_roof_tank.pdf new file mode 100644 index 0000000..e2f1e20 Binary files /dev/null and b/backend/data/knowledge/raw/case_studies/oisd_h2s_fatality_floating_roof_tank.pdf differ diff --git a/backend/data/knowledge/raw/case_studies/oisd_hydrocarbon_drain_line_fire.pdf b/backend/data/knowledge/raw/case_studies/oisd_hydrocarbon_drain_line_fire.pdf new file mode 100644 index 0000000..6a530c6 Binary files /dev/null and b/backend/data/knowledge/raw/case_studies/oisd_hydrocarbon_drain_line_fire.pdf differ diff --git a/backend/data/knowledge/raw/case_studies/oisd_naphtha_reboiler_furnace_fire.pdf b/backend/data/knowledge/raw/case_studies/oisd_naphtha_reboiler_furnace_fire.pdf new file mode 100644 index 0000000..6406abf Binary files /dev/null and b/backend/data/knowledge/raw/case_studies/oisd_naphtha_reboiler_furnace_fire.pdf differ diff --git a/backend/data/knowledge/raw/case_studies/oisd_wet_slop_pump_house_fire.pdf b/backend/data/knowledge/raw/case_studies/oisd_wet_slop_pump_house_fire.pdf new file mode 100644 index 0000000..8ba56d3 Binary files /dev/null and b/backend/data/knowledge/raw/case_studies/oisd_wet_slop_pump_house_fire.pdf differ diff --git a/backend/data/knowledge/raw/regulations/.gitkeep b/backend/data/knowledge/raw/regulations/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/backend/data/knowledge/raw/regulations/oisd_std_116_fire_protection.pdf b/backend/data/knowledge/raw/regulations/oisd_std_116_fire_protection.pdf new file mode 100644 index 0000000..a870bc1 Binary files /dev/null and b/backend/data/knowledge/raw/regulations/oisd_std_116_fire_protection.pdf differ diff --git a/backend/data/knowledge/raw/regulations/pngrb_haldia_refinery_naphtha_release.pdf b/backend/data/knowledge/raw/regulations/pngrb_haldia_refinery_naphtha_release.pdf new file mode 100644 index 0000000..20fd353 Binary files /dev/null and b/backend/data/knowledge/raw/regulations/pngrb_haldia_refinery_naphtha_release.pdf differ diff --git a/backend/data/knowledge/raw/synthetic_sops/emergency_evacuation_sop.md b/backend/data/knowledge/raw/synthetic_sops/emergency_evacuation_sop.md new file mode 100644 index 0000000..db1ea16 --- /dev/null +++ b/backend/data/knowledge/raw/synthetic_sops/emergency_evacuation_sop.md @@ -0,0 +1,57 @@ +# Synthetic Emergency Evacuation Procedure + +> SYNTHETIC PROTOTYPE SOP — NOT FOR REAL INDUSTRIAL USE + +## Purpose + +Provide a prototype response outline for a suspected fire, flash fire, +explosion risk, toxic-gas release, or other escalating refinery maintenance +condition with workers in the affected area. + +## Preconditions + +- The site emergency plan, alarm channels, evacuation routes, assembly areas, + access controls, and emergency contacts are known to the responsible team. +- Supervisors maintain a current account of workers, contractors, visitors, and + last known locations without delaying withdrawal. + +## Required checks + +Share the location, equipment, observed hazards, permit status, gas trends, +ventilation condition, isolation status, and last known worker conditions with +the emergency coordinator. Confirm accountability at the assembly area and +report missing or injured people through the site process. + +## Stop-work or suspension conditions + +Stop all affected work and suspend permits for an alarm, rising hydrocarbon gas, +active ignition source, failed ventilation, suspected release, incomplete +isolation, or an instruction from the emergency authority. Do not delay an +evacuation for another atmosphere test. + +## Immediate actions + +Warn nearby people without creating additional exposure, stop work if safe, +raise the site alarm, withdraw by the designated route, and muster. Do not +re-enter, retrieve tools, investigate, or attempt rescue without authorization +and the appropriate emergency capability. + +## Human approvals + +The incident commander or designated emergency authority directs evacuation, +accountability, controlled re-entry, permit suspension, and communications with +external responders. Workers may withdraw and report danger without waiting for +an automated recommendation. + +## Reauthorization conditions + +Re-entry requires a documented human declaration that the area is controlled, +verified isolation and ventilation, atmospheric retesting as applicable, +accountability, permit review, and approval of the incident commander and site +operations authority. + +## Limitations + +This synthetic outline is not an emergency plan and does not replace site +alarms, muster arrangements, rescue procedures, medical response, or approved +industrial operating instructions. diff --git a/backend/data/knowledge/raw/synthetic_sops/equipment_isolation_sop.md b/backend/data/knowledge/raw/synthetic_sops/equipment_isolation_sop.md new file mode 100644 index 0000000..74f174e --- /dev/null +++ b/backend/data/knowledge/raw/synthetic_sops/equipment_isolation_sop.md @@ -0,0 +1,56 @@ +# Synthetic Equipment Isolation Procedure + +> SYNTHETIC PROTOTYPE SOP — NOT FOR REAL INDUSTRIAL USE + +## Purpose + +Provide a prototype sequence for isolating a pump, valve, or connected system +before maintenance, especially when hydrocarbon release or hot work is possible. + +## Preconditions + +- The work party and operations representative agree on the equipment identity, + boundary, scope, and isolation plan. +- Process, electrical, hydraulic, pneumatic, thermal, stored-pressure, remote- + start, and connected-system energy sources are identified. +- The approved isolation, lockout, tagout, depressurization, draining, and + verification processes are available at the worksite. + +## Required checks + +Apply and document every required isolation and lock. Verify the expected +zero-energy condition using the approved method. Confirm drains, vents, +interlocks, bypasses, adjacent equipment, and backflow paths cannot re-energize +the boundary. Record equipment identifiers and responsible approvers. + +## Stop-work or suspension conditions + +Stop if an isolation is incomplete, a valve position is uncertain, pressure or +flow returns, equipment identification is unclear, a lock is missing, the scope +changes, or a worker is inside an unverified boundary. Treat an unverified +isolation as unsafe. + +## Immediate actions + +Stop the task, keep people clear of the boundary, do not manipulate an uncertain +valve or remove a lock, and notify operations and the supervisor. If release, +fire, or gas exposure is suspected, withdraw and use the site alarm or +evacuation process as directed. + +## Human approvals + +Authorized operations and maintenance roles must verify the isolation and +approve the work boundary. A safety representative or permit authority may +require additional controls. Software does not operate, isolate, or restart +equipment. + +## Reauthorization conditions + +Work may resume only after the isolation discrepancy is resolved, zero energy is +reverified, affected permits and gas tests are reviewed, the work party is +briefed, and authorized operations and maintenance roles approve continuation. + +## Limitations + +This synthetic checklist is not a replacement for a facility isolation +standard, lockout/tagout program, engineering review, or permit system. diff --git a/backend/data/knowledge/raw/synthetic_sops/gas_testing_sop.md b/backend/data/knowledge/raw/synthetic_sops/gas_testing_sop.md new file mode 100644 index 0000000..beadf96 --- /dev/null +++ b/backend/data/knowledge/raw/synthetic_sops/gas_testing_sop.md @@ -0,0 +1,57 @@ +# Synthetic Gas Testing and Atmosphere Monitoring Procedure + +> SYNTHETIC PROTOTYPE SOP — NOT FOR REAL INDUSTRIAL USE + +## Purpose + +Describe a prototype workflow for checking and trending the atmosphere before +and during maintenance, hot work, line breaking, or entry. The approved site +procedure and instrument instructions always control. + +## Preconditions + +- The tester is trained and authorized, and the instrument is suitable, + inspected, calibrated, and function-checked under the site process. +- The sampling plan covers the work location, low and high points where + relevant, likely release paths, and the effect of failed ventilation. +- Results can be recorded against the permit, equipment, location, timestamp, + and tester, with a reliable communication path to operations. + +## Required checks + +Check oxygen, flammable-gas indication, and toxic-gas indicators required by the +work scope. Compare readings with the approved site limits and trend them over +time. Treat rising LEL or hydrocarbon concentration, unexplained change, a +failed instrument, or a lost sample as an unsafe change requiring review. + +## Stop-work or suspension conditions + +Stop entry and hot work for an alarm, rising LEL, high H2S, low oxygen, +ventilation failure, loss of communications, an instrument fault, or any +reading that cannot be trusted. Suspend the affected permit until operations +and safety authorities determine the next step. + +## Immediate actions + +Stop work, warn nearby workers, withdraw people from the affected area when +conditions warrant, preserve the instrument record, and notify the supervisor +and operations representative. Use the site alarm and evacuation procedure if +directed; do not enter to investigate without authorization. + +## Human approvals + +The authorized tester reports results, but the responsible operations and safety +roles decide whether controls are adequate, a permit is suspended, or emergency +response is required. No automated reading authorizes a restart. + +## Reauthorization conditions + +Reauthorization requires the cause of the change to be reviewed, controls and +ventilation to be verified, a new or repeated atmosphere test to be recorded, +and the applicable permit to be approved again by authorized people. + +## Limitations + +This synthetic procedure does not set universal numerical thresholds, replace +instrument instructions or calibration programs, or serve as a confined-space, +hot-work, or emergency plan. diff --git a/backend/data/knowledge/raw/synthetic_sops/hot_work_sop.md b/backend/data/knowledge/raw/synthetic_sops/hot_work_sop.md new file mode 100644 index 0000000..5254109 --- /dev/null +++ b/backend/data/knowledge/raw/synthetic_sops/hot_work_sop.md @@ -0,0 +1,58 @@ +# Synthetic Hot Work Safety Procedure + +> SYNTHETIC PROTOTYPE SOP — NOT FOR REAL INDUSTRIAL USE + +## Purpose + +Provide a prototype evidence checklist for welding, cutting, grinding, or any +other ignition-producing task near refinery equipment. It is not a legal, +engineering, or operating standard. + +## Preconditions + +- An authorized supervisor has confirmed the exact work boundary, equipment, + active hot-work permit, fire watch, communications, and escape route. +- Operations has reviewed the isolation plan and known hydrocarbon release paths. +- The work party is briefed on stop-work authority and the site emergency plan. + +## Required checks + +- Test the atmosphere with an approved, suitable, in-calibration instrument and + record the time, location, tester, and result against the permit. +- Continue monitoring as required by the site procedure, including likely + release paths. Rising hydrocarbon readings, high H2S, low oxygen, or a failed + instrument requires immediate suspension and reassessment. +- Confirm combustible materials are controlled, the fire watch is available, + workers are accounted for, and access remains clear. + +## Stop-work or suspension conditions + +Suspend hot work immediately if the permit is suspended or expired, gas +conditions trend adversely, ventilation fails, isolation is incomplete or +uncertain, communications are lost, or the fire watch is unavailable. Do not +restart simply because a later reading appears better. + +## Immediate actions + +Stop the ignition source safely, warn the work party, notify operations and the +area supervisor, and move workers to the designated safe location when exposure +or fire risk warrants. Initiate the site alarm and evacuation process when +directed by the responsible emergency authority. + +## Human approvals + +Only the authorized supervisor, operations representative, and applicable safety +or emergency authority may decide whether controls are adequate, suspend the +permit, or approve a response. This prototype never issues an equipment command. + +## Reauthorization conditions + +Reauthorization requires a verified control plan, completed isolation review, +new atmosphere testing, restored ventilation where required, an available fire +watch, and documented human approval under the site permit process. + +## Limitations + +Do not use this synthetic prototype for real industrial work. It does not define +universal gas thresholds, ventilation performance, PPE, permit rules, or an +emergency response plan. diff --git a/backend/data/knowledge/raw/synthetic_sops/line_breaking_sop.md b/backend/data/knowledge/raw/synthetic_sops/line_breaking_sop.md new file mode 100644 index 0000000..cea39bd --- /dev/null +++ b/backend/data/knowledge/raw/synthetic_sops/line_breaking_sop.md @@ -0,0 +1,54 @@ +# Synthetic Line Breaking Procedure + +> SYNTHETIC PROTOTYPE SOP — NOT FOR REAL INDUSTRIAL USE + +## Purpose + +Give a prototype checklist for opening a process line, flange, drain, vessel +connection, or other boundary where containment may be released. + +## Preconditions + +- The line-breaking scope, equipment identity, contents, pressure state, and + isolation boundary are confirmed with operations. +- Required line-breaking and related permits are active, and a competent + supervisor is present. +- Depressurization, draining, purging, containment, PPE, exclusion, gas + testing, and emergency arrangements are reviewed. + +## Required checks + +Use the approved verification method before loosening the boundary. Check for +pressure, unexpected flow, liquid, vapor, rising LEL, H2S, and ventilation +conditions. Keep nonessential workers outside the controlled area and maintain +communications with operations. + +## Stop-work or suspension conditions + +Stop immediately for pressure, unexpected odor or release, rising LEL, high +H2S, low oxygen, incomplete isolation, loss of ventilation, changed contents, +or any work condition not covered by the permit. + +## Immediate actions + +Stop loosening the boundary, keep the opening closed if safe to do so, warn the +work party, withdraw from the affected area, and notify operations and the +supervisor. Initiate the site alarm or evacuation process when directed; do not +approach a release to investigate without authorization. + +## Human approvals + +Operations and safety or permit authorities decide whether the line is safe to +open, whether the permit must be suspended, and whether emergency response is +needed. Workers retain stop-work authority. + +## Reauthorization conditions + +Resume only after containment is restored or the revised control plan is +approved, isolation and depressurization are reverified, atmosphere testing is +repeated, and authorized operations and permit roles approve the changed scope. + +## Limitations + +This synthetic demonstration is not a universal line-breaking procedure and +does not define pressure, gas, PPE, isolation, or emergency thresholds. diff --git a/backend/data/knowledge/raw/synthetic_sops/ventilation_failure_sop.md b/backend/data/knowledge/raw/synthetic_sops/ventilation_failure_sop.md new file mode 100644 index 0000000..b1b1719 --- /dev/null +++ b/backend/data/knowledge/raw/synthetic_sops/ventilation_failure_sop.md @@ -0,0 +1,54 @@ +# Synthetic Ventilation Failure Response Procedure + +> SYNTHETIC PROTOTYPE SOP — NOT FOR REAL INDUSTRIAL USE + +## Purpose + +Describe a prototype response when fixed or temporary ventilation is unavailable +or ineffective during refinery maintenance. + +## Preconditions + +- The area, equipment, work scope, permit conditions, airflow path, and worker + locations are known to operations. +- Gas-testing equipment, communications, alarm routes, and designated safe + locations are available and understood by the work party. +- A qualified person has identified an approved contingency or work suspension + path; an improvised fan or open door is not assumed to be a control. + +## Required checks + +Confirm the ventilation failure, affected boundary, gas readings and trends, +possible release paths, worker count, and communication status. Review whether +an engineered control, safe alternate method, or emergency response is required. + +## Stop-work or suspension conditions + +Stop ignition-producing work and suspend affected permits for ventilation +failure, rising hydrocarbon concentration, high H2S, low oxygen, an alarm, or an +unverified airflow control. Keep work suspended if the control cannot be +verified. + +## Immediate actions + +Stop work, isolate ignition sources when safe, notify the area supervisor and +operations representative, account for workers, and move people to the +designated safe location when conditions warrant. Raise the site alarm and +evacuate when directed by the emergency authority. + +## Human approvals + +Operations, safety, and the responsible permit authority decide whether to +suspend permits, escalate to emergency response, or select an engineered +control. No software output authorizes a fan, valve, or other equipment command. + +## Reauthorization conditions + +Restart requires a verified ventilation control or approved alternate method, +new or repeated atmosphere checks, confirmed communications, worker briefing, +and permit reauthorization by the responsible human roles. + +## Limitations + +This synthetic procedure does not define universal ventilation performance, +exposure limits, gas thresholds, emergency routes, or a facility restart plan. diff --git a/backend/data/knowledge/vector_store/.gitkeep b/backend/data/knowledge/vector_store/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/backend/tests/test_rag.py b/backend/tests/test_rag.py new file mode 100644 index 0000000..5d481b4 --- /dev/null +++ b/backend/tests/test_rag.py @@ -0,0 +1,317 @@ +"""Offline tests for the refinery-safety knowledge-base foundation.""" + +import sys +from pathlib import Path +from types import SimpleNamespace + +import pytest +from pydantic import ValidationError + +from app.rag.chunker import chunk_pages +from app.rag.cli import ingest_documents +from app.rag.document_loader import DocumentLoadError, load_document +from app.rag.embedder import DeterministicEmbedder +from app.rag.metadata import enrich_chunks, load_manifest +from app.rag.models import DocumentPage, RetrievalQuery, RetrievalResult, SourceDocument +from app.rag.retriever import Retriever, risk_to_retrieval_query +from app.rag.text_cleaner import clean_pages, clean_text +from app.rag.vector_store import JsonVectorStore +from app.schemas import HazardCode, RiskEngineInput, RiskType + + +ROOT = Path(__file__).parents[2] +MANIFEST = ROOT / "backend/data/knowledge/manifests/documents.json" +KNOWLEDGE_ROOT = ROOT / "backend/data/knowledge" + + +def sample_page(text: str, page_number: int = 1) -> DocumentPage: + return DocumentPage( + document_id="DOC-1", + page_number=page_number, + text=text, + source_title="Local safety note", + source_path="fixture.md", + document_type="SOP", + authority="Test authority", + ) + + +def sample_risk() -> RiskEngineInput: + return RiskEngineInput( + alert_id="ALT-1", + timestamp="2026-07-16T09:30:00Z", + zone_id="ZONE_B", + equipment_ids=["P-101"], + risk_type="FIRE_EXPLOSION", + risk_score=0.9, + severity="CRITICAL", + predicted_incident="Hydrocarbon flash fire", + contributing_factors=[ + "RISING_LEL", + "HOT_WORK_ACTIVE", + "VENTILATION_FAILURE", + "WORKERS_PRESENT", + ], + sensor_evidence={"lel": 8.4}, + active_permit_ids=["PTW-1"], + ) + + +def test_markdown_loading_and_provenance(tmp_path: Path) -> None: + path = tmp_path / "note.md" + path.write_text("# Heading\n\nMust not start work without a permit.", encoding="utf-8") + document = SourceDocument( + document_id="DOC-1", + title="Local note", + document_type="SOP", + source_path=str(path), + ) + + pages = load_document(document) + + assert len(pages) == 1 + assert pages[0].page_number == 1 + assert pages[0].source_path == str(path) + + +def test_text_loading_and_empty_file_handling(tmp_path: Path) -> None: + path = tmp_path / "note.txt" + path.write_text("A text source", encoding="utf-8") + document = SourceDocument(document_id="DOC-1", title="Text", document_type="SOP", source_path=str(path)) + assert load_document(document)[0].text == "A text source" + + empty = tmp_path / "empty.txt" + empty.touch() + with pytest.raises(DocumentLoadError): + load_document(document.model_copy(update={"source_path": str(empty)})) + + whitespace = tmp_path / "whitespace.txt" + whitespace.write_text(" \n\t", encoding="utf-8") + with pytest.raises(DocumentLoadError, match="contains no text"): + load_document(document.model_copy(update={"source_path": str(whitespace)})) + + +def test_pdf_loading_preserves_page_numbers_without_network(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None: + class FakePage: + def __init__(self, text: str) -> None: + self._text = text + + def extract_text(self) -> str: + return self._text + + class FakeReader: + def __init__(self, _: str) -> None: + self.pages = [FakePage("Page one"), FakePage(""), FakePage("Page two")] + + monkeypatch.setitem(sys.modules, "pypdf", SimpleNamespace(PdfReader=FakeReader)) + path = tmp_path / "note.pdf" + path.write_bytes(b"%PDF-test") + document = SourceDocument(document_id="DOC-1", title="PDF", document_type="REGULATION", source_path=str(path)) + + pages = load_document(document) + + assert [page.page_number for page in pages] == [1, 3] + + +def test_unsupported_and_missing_file_handling(tmp_path: Path) -> None: + document = SourceDocument(document_id="DOC-1", title="Unknown", document_type="SOP", source_path=str(tmp_path / "a.docx")) + with pytest.raises(DocumentLoadError, match="does not exist"): + load_document(document) + path = tmp_path / "a.docx" + path.write_text("content", encoding="utf-8") + with pytest.raises(DocumentLoadError, match="Unsupported"): + load_document(document.model_copy(update={"source_path": str(path)})) + + +def test_cleaning_is_conservative_and_removes_repeated_page_lines() -> None: + assert clean_text("must not\x00\n\n\nbe removed") == "must not\n\nbe removed" + pages = clean_pages([sample_page("HEADER\n\nKeep this must not clause\nFOOTER", 1), sample_page("HEADER\n\nAnother clause\nFOOTER", 2)]) + assert all("HEADER" not in page.text and "FOOTER" not in page.text for page in pages) + assert "must not" in pages[0].text + + +def test_chunk_ids_are_deterministic_and_overlap_is_present() -> None: + text = " ".join(f"word{i}" for i in range(30)) + first = chunk_pages([sample_page(text)], max_tokens=10, overlap_tokens=3) + second = chunk_pages([sample_page(text)], max_tokens=10, overlap_tokens=3) + + assert [chunk.chunk_id for chunk in first] == [chunk.chunk_id for chunk in second] + assert len(first) > 1 + assert set(first[0].text.split()[-3:]).intersection(first[1].text.split()[:3]) + assert all(chunk.text.strip() for chunk in first) + + +def test_manifest_parsing_and_metadata_keyword_tagging() -> None: + entries = load_manifest(MANIFEST) + assert len(entries) == 18 + assert sum(entry.is_synthetic for entry in entries) == 6 + assert all(entry.local_path for entry in entries) + assert all(entry.local_path.startswith("raw/") for entry in entries) + chunks = enrich_chunks( + chunk_pages( + [ + sample_page( + "Hot work permit is active. Rising LEL and ventilation failure are observed while workers are present." + ) + ] + ) + ) + assert "RISING_LEL" in {value.value for value in chunks[0].hazard_codes} + assert "HOT_WORK" in {value.value for value in chunks[0].permit_types} + + +def test_retrieval_models_reject_invalid_scores() -> None: + with pytest.raises(ValidationError): + RetrievalResult( + chunk_id="C-1", + text="A useful evidence passage.", + similarity_score=1.1, + source_title="Source", + page_start=1, + page_end=1, + ) + with pytest.raises(ValidationError): + RetrievalQuery(query_text="valid query", top_k=21) + + +def test_vector_store_insertion_retrieval_and_filters(tmp_path: Path) -> None: + pages = [ + sample_page("Hot work permit controls require gas testing and a fire watch."), + DocumentPage( + document_id="DOC-2", + page_number=1, + text="A historical incident involved an electrical failure.", + source_title="Incident record", + source_path="incident.md", + document_type="HISTORICAL_INCIDENT", + ), + ] + chunks = [] + for page_group in ([pages[0]], [pages[1]]): + chunks.extend(enrich_chunks(chunk_pages(page_group))) + store = JsonVectorStore(tmp_path / "store.json") + embedder = DeterministicEmbedder() + store.add(chunks, embedder.embed([chunk.text for chunk in chunks])) + retriever = Retriever(store, embedder) + + results = retriever.retrieve("hot work gas testing", mode="regulations_and_sops") + + assert results + assert all(result.metadata["document_type"] == "SOP" for result in results) + assert "gas testing" in results[0].text.lower() + risk_results = retriever.retrieve( + RetrievalQuery(query_text="fire controls", risk_types=[RiskType.FIRE_EXPLOSION]), + mode="all", + ) + assert risk_results + assert all("FIRE_EXPLOSION" in result.metadata["risk_types"] for result in risk_results) + assert results[0].document_type == "SOP" + assert results[0].is_synthetic is False + + hazard_results = retriever.retrieve( + RetrievalQuery( + query_text="ventilation and rising gas", + hazard_codes=[HazardCode.HOT_WORK_ACTIVE], + ), + mode="all", + ) + assert hazard_results + assert all("HOT_WORK_ACTIVE" in result.metadata["hazard_codes"] for result in hazard_results) + + +def test_fake_embedding_is_deterministic_and_store_validates_dimension(tmp_path: Path) -> None: + embedder = DeterministicEmbedder(dimension=32) + assert embedder.embed(["same text"])[0] == embedder.embed(["same text"])[0] + chunk = enrich_chunks(chunk_pages([sample_page("A useful gas testing control passage.")]))[0] + store = JsonVectorStore(tmp_path / "dimension-store.json") + store.add([chunk], embedder.embed([chunk.text])) + assert store.embedding_dimension == 32 + with pytest.raises(ValueError, match="does not match store dimension"): + store.add([chunk], [[0.0] * 16]) + + +def test_vector_store_persists_and_repeated_ingestion_replaces_records(tmp_path: Path) -> None: + chunks = enrich_chunks(chunk_pages([sample_page("A persistent safety evidence passage.")])) + embedder = DeterministicEmbedder() + path = tmp_path / "persistent-store.json" + store = JsonVectorStore(path) + vectors = embedder.embed([chunk.text for chunk in chunks]) + store.add(chunks, vectors) + store.add(chunks, vectors) + assert store.count == 1 + reopened = JsonVectorStore(path) + assert reopened.count == 1 + assert Retriever(reopened, embedder).retrieve("persistent safety evidence") + + +def test_risk_engine_input_becomes_retrieval_query() -> None: + query = risk_to_retrieval_query(sample_risk()) + assert "pump station" in query.query_text + assert "hot work is active" in query.query_text + assert query.risk_types[0].value == "FIRE_EXPLOSION" + + +def test_synthetic_hot_work_retrieval(tmp_path: Path) -> None: + entries = load_manifest(MANIFEST) + chunks = [] + for entry in entries: + if not entry.is_synthetic: + continue + pages = clean_pages(load_document(entry.to_source_document(), KNOWLEDGE_ROOT)) + chunks.extend(enrich_chunks(chunk_pages(pages))) + embedder = DeterministicEmbedder() + store = JsonVectorStore(tmp_path / "synthetic-store.json") + store.add(chunks, embedder.embed([chunk.text for chunk in chunks])) + + results = Retriever(store, embedder).retrieve(risk_to_retrieval_query(sample_risk()), mode="regulations_and_sops") + + assert results + assert any("hot work" in result.text.lower() for result in results) + assert all(result.source_title for result in results) + + +def test_synthetic_scenario_retrieves_relevant_sops(tmp_path: Path) -> None: + entries = [entry for entry in load_manifest(MANIFEST) if entry.is_synthetic] + chunks = [] + for entry in entries: + chunks.extend(enrich_chunks(chunk_pages(clean_pages(load_document(entry.to_source_document(), KNOWLEDGE_ROOT))))) + embedder = DeterministicEmbedder() + store = JsonVectorStore(tmp_path / "scenario-store.json") + store.add(chunks, embedder.embed([chunk.text for chunk in chunks])) + query = ( + "Hydrocarbon gas is rising near Pump P-101 while a hot-work permit is active. " + "Ventilation has failed and workers are present in Zone B." + ) + results = Retriever(store, embedder).retrieve( + RetrievalQuery(query_text=query, document_types=["SOP"], top_k=6), + mode="all", + ) + titles = {result.source_title for result in results} + assert { + "Synthetic Hot Work Safety Procedure", + "Synthetic Gas Testing and Atmosphere Monitoring Procedure", + "Synthetic Ventilation Failure Response Procedure", + "Synthetic Emergency Evacuation Procedure", + } <= titles + + +def test_incremental_cli_ingestion_reports_and_is_idempotent(tmp_path: Path) -> None: + manifest = tmp_path / "manifests" / "documents.json" + manifest.parent.mkdir(parents=True) + knowledge_root = manifest.parent.parent + source = knowledge_root / "raw" / "note.md" + source.parent.mkdir(parents=True) + source.write_text("# Local SOP\n\nA useful gas testing control passage.", encoding="utf-8") + manifest.write_text( + "[{\"document_id\":\"DOC-LOCAL\",\"title\":\"Local SOP\"," + "\"document_type\":\"SOP\",\"local_path\":\"raw/note.md\"," + "\"allowed_for_public_repo\":true,\"is_synthetic\":true," + "\"processing_status\":\"ready\"}]", + encoding="utf-8", + ) + store_path = tmp_path / "vector" / "chunks.json" + first = ingest_documents(manifest, store_path, rebuild=True, embedder_name="deterministic") + second = ingest_documents(manifest, store_path, rebuild=False, embedder_name="deterministic") + assert first.documents_loaded == second.documents_loaded == 1 + assert first.chunks_created == second.chunks_created == 1 + assert second.persisted_chunks == 1 diff --git a/backend/tests/test_schemas.py b/backend/tests/test_schemas.py new file mode 100644 index 0000000..9786f50 --- /dev/null +++ b/backend/tests/test_schemas.py @@ -0,0 +1,217 @@ +"""Validation tests for the shared SpecGuard service contracts.""" + +import json +from datetime import datetime, timezone +from pathlib import Path + +import pytest +from pydantic import BaseModel, ValidationError + +from app.schemas import ( + ExtractedSafetyEvent, + HistoricalIncident, + IncidentSearchQuery, + RecommendedAction, + RiskEngineInput, + SafetyIntelligenceResponse, + ShiftLogEntry, +) + + +EXAMPLES_DIR = Path(__file__).parents[2] / "docs" / "examples" +NOW = datetime(2026, 7, 16, 9, 30, tzinfo=timezone.utc) + + +def valid_risk_input() -> dict[str, object]: + return { + "alert_id": "ALT-2026-001", + "timestamp": NOW, + "zone_id": "ZONE_B", + "equipment_ids": ["P-101"], + "risk_type": "FIRE_EXPLOSION", + "risk_score": 0.91, + "severity": "CRITICAL", + "predicted_incident": "Hydrocarbon flash fire near pump P-101", + "contributing_factors": [ + "RISING_LEL", + "HOT_WORK_ACTIVE", + "VENTILATION_FAILURE", + "WORKERS_PRESENT", + ], + "sensor_evidence": {"hc_gas_lel": {"value": 8.4, "unit": "%LEL"}}, + "active_permit_ids": ["PTW-HW-204"], + "maintenance_event_ids": ["MNT-VENT-031"], + "shift_log_ids": ["LOG-NIGHT-118"], + "cctv_event_ids": ["CCTV-ZB-778"], + "estimated_lead_time_minutes": 12.0, + "model_confidence": 0.88, + } + + +def test_valid_risk_engine_input_is_accepted() -> None: + model = RiskEngineInput.model_validate(valid_risk_input()) + + assert model.alert_id == "ALT-2026-001" + assert model.zone_id.value == "ZONE_B" + + +def test_risk_score_above_one_is_rejected() -> None: + payload = valid_risk_input() + payload["risk_score"] = 1.01 + + with pytest.raises(ValidationError): + RiskEngineInput.model_validate(payload) + + +def test_negative_lead_time_is_rejected() -> None: + payload = valid_risk_input() + payload["estimated_lead_time_minutes"] = -0.1 + + with pytest.raises(ValidationError): + RiskEngineInput.model_validate(payload) + + +def test_invalid_enum_value_is_rejected() -> None: + payload = valid_risk_input() + payload["severity"] = "EXTREME" + + with pytest.raises(ValidationError): + RiskEngineInput.model_validate(payload) + + +def test_extraction_confidence_above_one_is_rejected() -> None: + with pytest.raises(ValidationError): + ExtractedSafetyEvent( + source_log_id="LOG-1", + severity="HIGH", + confidence=1.1, + summary="Hydrocarbon readings are trending upward.", + requires_follow_up=True, + ) + + +def test_empty_required_text_is_rejected() -> None: + with pytest.raises(ValidationError): + ShiftLogEntry( + log_id="LOG-1", + timestamp=NOW, + shift_id="NIGHT-1", + author_role="Operator", + raw_text=" ", + acknowledged=False, + resolved=False, + ) + + +def test_valid_shift_log_is_accepted() -> None: + model = ShiftLogEntry( + log_id="LOG-1", + timestamp=NOW, + shift_id="NIGHT-1", + author_role="Console Operator", + zone_id="ZONE_B", + equipment_ids=["P-101"], + raw_text="Hydrocarbon smell reported near P-101 during rounds.", + acknowledged=True, + resolved=False, + ) + + assert model.equipment_ids == ["P-101"] + + +def test_valid_extracted_safety_event_is_accepted() -> None: + model = ExtractedSafetyEvent( + source_log_id="LOG-1", + zone_id="ZONE_B", + equipment_ids=["P-101"], + hazards=["RISING_LEL"], + severity="HIGH", + confidence=0.87, + summary="Possible hydrocarbon accumulation near pump P-101.", + requires_follow_up=True, + ) + + assert model.extraction_method == "RULE_LLM_HYBRID" + + +def test_valid_historical_incident_is_accepted() -> None: + model = HistoricalIncident( + incident_id="INC-001", + title="Pump seal leak ignited during hot work", + risk_type="FIRE_EXPLOSION", + severity="CRITICAL", + summary="A seal leak accumulated vapour near an ignition source.", + is_synthetic=True, + ) + + assert model.industry == "PETROLEUM_REFINERY" + + +@pytest.mark.parametrize("top_k", [0, 21]) +def test_invalid_top_k_is_rejected(top_k: int) -> None: + with pytest.raises(ValidationError): + IncidentSearchQuery(top_k=top_k) + + +def test_complete_response_serializes_to_json() -> None: + payload = json.loads( + (EXAMPLES_DIR / "safety_intelligence_response.json").read_text() + ) + model = SafetyIntelligenceResponse.model_validate(payload) + serialized = json.loads(model.model_dump_json()) + + assert serialized["alert_id"] == "ALT-2026-001" + assert serialized["recommended_actions"][0]["status"] == "PROPOSED" + assert serialized["requires_human_review"] is True + + +def test_unknown_extra_fields_are_rejected() -> None: + payload = valid_risk_input() + payload["unexpected_control_command"] = "START_PUMP" + + with pytest.raises(ValidationError): + RiskEngineInput.model_validate(payload) + + +def test_recommended_action_defaults_to_human_approval() -> None: + action = RecommendedAction( + action_id="ACT-001", + title="Review suspension of hot work", + description="A supervisor should verify conditions and apply the site SOP.", + priority=10, + status="PROPOSED", + target_role="Area Supervisor", + ) + + assert action.requires_human_approval is True + + +def test_intelligence_response_defaults_to_human_review() -> None: + response = SafetyIntelligenceResponse( + intelligence_id="INT-001", + alert_id="ALT-001", + generated_at=NOW, + executive_summary="A compound flash-fire risk exists in Zone B.", + risk_explanation="Rising hydrocarbons coincide with an ignition source.", + intelligence_confidence=0.8, + insufficient_evidence=False, + ) + + assert response.requires_human_review is True + + +@pytest.mark.parametrize( + ("filename", "schema"), + [ + ("risk_engine_input.json", RiskEngineInput), + ("safety_intelligence_response.json", SafetyIntelligenceResponse), + ("shift_log_entry.json", ShiftLogEntry), + ("extracted_safety_event.json", ExtractedSafetyEvent), + ("historical_incident.json", HistoricalIncident), + ], +) +def test_documentation_example_validates( + filename: str, + schema: type[BaseModel], +) -> None: + schema.model_validate_json((EXAMPLES_DIR / filename).read_text()) diff --git a/docs/RAG_KNOWLEDGE_BASE.md b/docs/RAG_KNOWLEDGE_BASE.md new file mode 100644 index 0000000..a06fc4c --- /dev/null +++ b/docs/RAG_KNOWLEDGE_BASE.md @@ -0,0 +1,192 @@ +# SpecGuard RAG Knowledge Base + +This is the Task 2 retrieval layer for the refinery-safety prototype. It stops +at evidence retrieval: it does not generate final recommendations, call an +LLM, or control refinery equipment. + +## What RAG is and why SpecGuard uses it + +Retrieval-augmented generation (RAG) first finds relevant passages in a known +collection and may later give those passages to a language model. Retrieval is +evidence, not a validated answer. SpecGuard uses it to connect sensor, permit, +and risk signals to procedures, regulations, and incident records that a human +reviewer can inspect and cite. + +```text +authorized local documents + -> loader -> conservative cleaner -> chunker -> safety metadata + -> local embeddings -> persistent JSON vector store -> filtered retrieval +``` + +## Folder structure + +```text +backend/app/rag/ +├── models.py # internal page, chunk, manifest, and result models +├── document_loader.py # PDF, Markdown, and plain-text loading +├── text_cleaner.py # conservative layout cleanup +├── chunker.py # deterministic section-aware chunks +├── metadata.py # manifest parsing and keyword tags +├── embedder.py # deterministic and local Sentence Transformers modes +├── vector_store.py # persistent JSON vector store +├── retriever.py # filtering and RiskEngineInput conversion +└── cli.py # inspect, ingest, and query commands + +backend/data/knowledge/ +├── raw/case_studies/ +├── raw/regulations/ +├── raw/synthetic_sops/ +├── manifests/documents.json +├── processed/ +└── vector_store/ +``` + +The manifest currently contains six public synthetic SOPs and twelve local PDF +entries. The PDFs use their real local filenames, have no fabricated URLs, and +are marked `pending_review` and not public-repository safe until authorization +and provenance are verified. + +## Loading documents + +The loader accepts `.pdf`, `.md`, and `.txt`. Markdown and text become page 1; +PDF pages retain their page numbers. Source document ID, title, local path, +URL, authority, publication metadata, version, tags, document type, and +synthetic status are carried into pages, chunks, and result metadata. + +Missing files, empty files, unsupported extensions, decode errors, PDF parser +errors, and PDFs with no extractable text are reported as skips. OCR is not +attempted. A scanned PDF therefore needs an authorized, separate OCR workflow. + +## Conservative cleaning + +Cleaning removes null bytes, normalizes line endings and repeated whitespace, +collapses excessive blank lines, and joins only a clear lowercase word break at +a line boundary. An exact first/last line repeated across pages is treated as a +header/footer. Headings, numbered clauses, bullets, equipment identifiers, +units, and negations such as `not`, `must not`, and `prohibited` are preserved. + +Aggressive rewriting is dangerous for safety documents: changing one negation, +unit, identifier, or clause can reverse the meaning of a control. The cleaner +therefore does not paraphrase or summarize. + +## Chunking and metadata + +The chunker uses word count as a transparent token approximation. Defaults are +700 words per chunk and 120 words of overlap, both configurable. It recognizes +Markdown and numbered headings, emits no empty or meaningless tiny chunks, and +stores deterministic IDs, page ranges, section hints, document metadata, and +source provenance. Rebuilding the same source produces the same chunk IDs. + +`metadata.py` applies deterministic keyword mappings; no LLM is used for +ingestion. It reuses the controlled `RiskType`, `HazardCode`, and `PermitType` +enums from `app.schemas.common`. Supported mappings include `FIRE_EXPLOSION`, +`TOXIC_GAS_EXPOSURE`, `OXYGEN_DEFICIENCY`, `OVERPRESSURE`, +`EQUIPMENT_FAILURE`, `RISING_LEL`, `HIGH_H2S`, `LOW_OXYGEN`, +`PRESSURE_RISING`, `VENTILATION_FAILURE`, `INCOMPLETE_ISOLATION`, +`HOT_WORK_ACTIVE`, `CONFINED_SPACE_ACTIVE`, `WORKERS_PRESENT`, +`OVERDUE_MAINTENANCE`, and the requested permit types. + +## Embeddings and vector retrieval + +`DeterministicEmbedder` is a hash-based fake embedder for tests and repeatable +offline demos. It never downloads a model. `SentenceTransformerEmbedder` is the +real local adapter and defaults to +`sentence-transformers/all-MiniLM-L6-v2`; its model name is configurable with +`SPECGUARD_EMBEDDING_MODEL` or the adapter constructor. It batches text, +normalizes vectors, and reports a helpful error if the package or model is not +available. No API key or paid API is required. + +The lightweight persistent backend is an inspectable JSON file. It supports +add, rebuild, persistent reload, document replacement, top-k cosine search, +document-type/risk/hazard/permit filters, dimension validation, and deterministic +tie ordering. Reprocessing a document replaces its records, so repeated +ingestion does not create uncontrolled duplicate chunks. Generated indexes are +ignored by Git. + +Every result includes chunk ID, text, score, title, document type, source path +or URL, page range, section when available, synthetic status, and the complete +chunk metadata dictionary. + +## Adding an authorized PDF + +1. Obtain the document through a permitted, authorized process; do not + automatically download standards or incident reports. +2. Place it in `backend/data/knowledge/raw/regulations/` or + `backend/data/knowledge/raw/case_studies/`. +3. Add a manifest entry using a path relative to + `backend/data/knowledge/`, for example + `raw/regulations/oisd_std_116_fire_protection.pdf`. +4. Use the real title and filename, a real source URL only when verified, the + correct authority/publication/version values, and an optional SHA-256 + checksum. Use `pending_review` or `missing` when it is not ready; never + pretend a missing file was loaded. +5. Confirm licensing and set `allowed_for_public_repo` correctly. A local file + marked false must not be committed or redistributed. +6. Ingest, inspect the report, and verify extracted pages and attribution. + +## Verified commands + +From the repository root, the offline deterministic commands are: + +```bash +PYTHONPATH=backend .venv/bin/python -m app.rag.cli inspect +PYTHONPATH=backend .venv/bin/python -m app.rag.cli ingest --rebuild +PYTHONPATH=backend .venv/bin/python -m app.rag.cli query \ + "Hydrocarbon gas is rising near Pump P-101 while a hot-work permit is active. Ventilation has failed and workers are present in Zone B." \ + --mode all --top-k 6 --document-type SOP +``` + +The equivalent commands from `backend/` are: + +```bash +cd backend +PYTHONPATH=. ../.venv/bin/python -m app.rag.cli inspect +PYTHONPATH=. ../.venv/bin/python -m app.rag.cli ingest --rebuild +``` + +`inspect` displays manifest entries, statuses, persisted chunk count, and +embedding dimension. `ingest` displays documents loaded/skipped, missing files, +extraction failures, chunks created/indexed, and every skip reason. Use +`--embedder sentence-transformers` before the subcommand when the package and +model are installed locally. Query also supports `--mode` values +`regulations_and_sops`, `similar_incidents`, or `all`, plus repeatable +`--risk-type`, `--hazard`, `--permit-type`, and `--document-type` filters. + +## RiskEngineInput integration + +`risk_to_retrieval_query` deterministically converts the existing public +`RiskEngineInput` into a `RetrievalQuery`. It includes the predicted incident, +zone, hazard wording, and requests for stop-work precautions, gas testing, +permit suspension, isolation verification, evacuation, reauthorization, and +similar incidents. It also carries controlled risk, hazard, and hot-work permit +filters. No LLM is involved. + +```python +from app.rag.retriever import risk_to_retrieval_query + +retrieval_query = risk_to_retrieval_query(risk_engine_input) +results = retriever.retrieve(retrieval_query, mode="regulations_and_sops") +``` + +This task does not map results into a final `SafetyIntelligenceResponse` yet. +Future actions must remain advisory, human-approved, and explicitly supported +by checked evidence. + +## Tests, limitations, and safety + +Run the complete offline suite with: + +```bash +.venv/bin/python -m pytest +``` + +Tests use local fixtures and deterministic embeddings; they require no internet, +model download, API key, or real OISD PDF. The prototype has no OCR, no +validated industrial thresholds, no guarantee that keyword tags are complete, +and no guarantee that a retrieved passage is current or sufficient. Synthetic +SOPs are not approved procedures. Never use this repository to direct real +industrial work, bypass human approvals, or operate equipment. + +Respect copyright, licensing, and access restrictions for every local source. +Do not commit `.env`, model caches, generated indexes, restricted standards, or +incident reports without authorization. diff --git a/docs/SCHEMA_ARCHITECTURE.md b/docs/SCHEMA_ARCHITECTURE.md new file mode 100644 index 0000000..68a590e --- /dev/null +++ b/docs/SCHEMA_ARCHITECTURE.md @@ -0,0 +1,409 @@ +# SpecGuard Schema Architecture + +This guide explains the shared data contracts in `backend/app/schemas`. It is +written for contributors who know basic Python but are new to Pydantic and +FastAPI request/response models. + +## 1. What a schema is + +A schema defines the permitted shape of data. It names each field, states its +Python type, declares whether it is required, and can place rules on values. For +example, `risk_score: float` says the value is numeric, while Pydantic's +`ge=0.0, le=1.0` rules say it must be between zero and one. + +[Pydantic](https://docs.pydantic.dev/) turns incoming Python dictionaries or +JSON into typed Python objects. It rejects missing fields, wrong types, invalid +enum values, values outside configured ranges, blank required text, and unknown +fields in the new SpecGuard contracts. + +Schemas are **not database tables**. They do not save, update, or query data. +They are also not services: a schema does not read a sensor, retrieve a document, +call an LLM, or decide whether a refinery operation is safe. It validates data at +the boundary between components. + +## 2. Why SpecGuard needs schemas + +SpecGuard joins data generated by different people and services. Without a +contract, one component may emit `riskScore`, another may expect `risk_score`, +and a third may treat a percentage as a number from 0 to 100 instead of 0 to 1. +These mistakes are particularly dangerous when a dashboard is summarizing +compound safety risk. + +The main integration flow is: + +```text +Digital twin and data simulator + ↓ +Sensor, permit, maintenance and CCTV data + ↓ +Compound-risk engine + ↓ +RiskEngineInput + ↓ +NLP, incident retrieval and RAG pipeline + ↓ +SafetyIntelligenceResponse + ↓ +FastAPI backend + ↓ +Frontend dashboard +``` + +`RiskEngineInput` and `SafetyIntelligenceResponse` are stable handoff points. +The implementation inside either pipeline may change without forcing the other +team to depend on its private classes or internal dictionaries. + +## 3. Why enums are used + +A permit type could otherwise arrive as `Hot Work`, `hot_work`, `HOT-WORK`, or +`HOT_WORK`. Those strings look equivalent to a person but are different values +to Python, a database filter, and JavaScript. The shared `PermitType.HOT_WORK` +member always serializes as `"HOT_WORK"`. + +The same rule applies to severity, zones, risks, hazards, statuses, and evidence +types. Controlled values make joins and frontend filters reliable and cause a +misspelling to fail validation early. New services should import the enums from +`app.schemas` instead of defining local copies. + +The current digital twin predates this contract and emits values such as +`Zone_B`, lowercase permit types, and a different Zone B/Zone C area mapping. +An ingestion adapter must translate those simulator values to the canonical API +values. The existing simulator and legacy schemas were deliberately not changed +silently because that would alter current generated datasets. + +## 4. Explanation of every schema + +### `ShiftLogEntry` + +The shift-log service or simulator creates this model. The NLP extraction +pipeline consumes it. It preserves the operator's original wording for audit, +reprocessing, and comparison with extracted facts. + +```python +ShiftLogEntry( + log_id="LOG-118", + timestamp="2026-07-16T09:24:00Z", + shift_id="NIGHT-01", + author_role="Console Operator", + raw_text="Hydrocarbon odour reported near pump P-101.", + acknowledged=True, + resolved=False, +) +``` + +### `ExtractedSafetyEvent` + +Ashish's NLP pipeline creates this model from a `ShiftLogEntry`. The risk engine, +retrieval pipeline, and backend can consume it. It separates machine-readable +hazards from raw human text and records extraction confidence and method. + +```python +ExtractedSafetyEvent( + source_log_id="LOG-118", + hazards=["RISING_LEL", "UNRESOLVED_SHIFT_OBSERVATION"], + severity="HIGH", + confidence=0.89, + summary="Possible hydrocarbon accumulation requires follow-up.", + requires_follow_up=True, +) +``` + +### `HistoricalIncident` + +An incident ingestion/normalization job creates this model. Incident retrieval +and RAG consume it. It puts differently sourced incident reports into one shape, +including causes, consequences, prevention, provenance, and an explicit +`is_synthetic` label. + +```python +HistoricalIncident( + incident_id="INC-SYN-014", + title="Pump seal vapour ignited during hot work", + risk_type="FIRE_EXPLOSION", + severity="CRITICAL", + summary="Leaked vapour accumulated after ventilation failed.", + is_synthetic=True, +) +``` + +### `NearMissRecord` + +An operations reporting or ingestion service creates this model. Retrieval and +prevention analytics consume it. A near miss deserves its own model because no +injury may have occurred even though the *potential* severity was critical. +Near misses expose early warning patterns before they become incidents. + +```python +NearMissRecord( + near_miss_id="NM-042", + title="Hot work stopped after rising gas trend", + risk_type="FIRE_EXPLOSION", + potential_severity="CRITICAL", + description="A supervisor stopped work before the LEL alarm activated.", + immediate_actions=["Hot-work permit was suspended"], + is_synthetic=False, +) +``` + +### `IncidentSearchQuery` + +The risk/RAG pipeline or an API client creates this model. The incident search +service consumes it. It combines free text with structured filters and limits +result size to between 1 and 20. + +```python +IncidentSearchQuery( + risk_type="FIRE_EXPLOSION", + hazards=["RISING_LEL", "HOT_WORK_ACTIVE"], + natural_language_query="pump seal leak with failed ventilation", + top_k=5, +) +``` + +Production retrieval should eventually combine semantic similarity from an +embedding/vector search with metadata filters such as risk type, zone, permit, +equipment, and hazard. Semantic search alone may return textually similar but +operationally irrelevant cases; metadata filters alone may miss differently +worded cases. + +### `SimilarIncident` + +Ashish's incident retrieval pipeline creates this compact result. The RAG +pipeline, backend, and frontend consume it. It carries only the history and +provenance required to explain similarity and prevention lessons. + +```python +SimilarIncident( + incident_id="INC-SYN-014", + title="Pump seal vapour ignited during hot work", + similarity_score=0.89, + shared_hazards=["RISING_LEL", "HOT_WORK_ACTIVE"], + summary="Leaked vapour accumulated and found an ignition source.", +) +``` + +### `RiskEngineInput` + +Rex's compound-risk engine creates this model. Ashish's NLP, retrieval, and RAG +pipeline consumes it. This is the central inbound intelligence contract and +prevents those two components from depending on hidden implementation details. + +```python +RiskEngineInput( + alert_id="ALT-001", + timestamp="2026-07-16T09:30:00Z", + zone_id="ZONE_B", + equipment_ids=["P-101"], + risk_type="FIRE_EXPLOSION", + risk_score=0.91, + severity="CRITICAL", + predicted_incident="Hydrocarbon flash fire near pump P-101", + contributing_factors=["RISING_LEL", "HOT_WORK_ACTIVE"], + sensor_evidence={"hc_gas_lel": {"value": 8.4, "unit": "%LEL"}}, +) +``` + +`sensor_evidence` is deliberately `dict[str, Any]`. Pressure, flow, gas, +temperature, vibration, alarm state, and trend evidence do not share one honest +physical shape. More specific sensor-family schemas can be added as integration +stabilizes instead of pretending every variable came from one sensor. + +### `EvidenceReference` + +The RAG/retrieval pipeline creates this model. Recommendations, the backend, and +the frontend consume it. The response stores an excerpt and source pointer, not +a duplicate of an entire SOP, regulation, incident, or sensor history. Actions +refer to `evidence_id` values. + +```python +EvidenceReference( + evidence_id="EVD-SOP-001", + evidence_type="SOP", + title="Hot-work atmospheric safety procedure", + excerpt="Supervisory review is required when conditions change.", + source_name="Refinery SOP Library", + page_number=12, + relevance_score=0.92, +) +``` + +### `RecommendedAction` + +The safety-intelligence pipeline proposes this model. Qualified supervisors and +operators consume it through the backend/frontend workflow. It is advisory: +`requires_human_approval` defaults to `True`, and the contract contains no field +for an LLM to directly control a valve, pump, permit, alarm, or shutdown system. + +```python +RecommendedAction( + action_id="ACT-001", + title="Review suspension of hot work", + description="The supervisor should verify conditions and apply the SOP.", + priority=10, + status="PROPOSED", + target_role="Area Supervisor", + supporting_evidence_ids=["EVD-SOP-001"], +) +``` + +### `SafetyIntelligenceResponse` + +Ashish's combined NLP/retrieval/RAG pipeline creates this final response. The +FastAPI backend validates and returns it; Rajath's frontend renders its JSON. +It groups the explanation, proposed actions, evidence, similar incidents, +confidence, limitations, and explicit human-review status. + +```python +SafetyIntelligenceResponse( + intelligence_id="INT-001", + alert_id="ALT-001", + generated_at="2026-07-16T09:30:08Z", + executive_summary="A compound flash-fire risk exists in Zone B.", + risk_explanation="Rising hydrocarbon levels coincide with active hot work.", + intelligence_confidence=0.90, + insufficient_evidence=False, +) +``` + +## 5. Team ownership and integration + +### Rex + +Rex produces `RiskEngineInput` from the compound-risk engine. The object should +be validated at the producer boundary before it is sent downstream. + +### Ashish + +Ashish consumes `RiskEngineInput` and produces `ExtractedSafetyEvent`, +`SimilarIncident`, and `SafetyIntelligenceResponse`. Evidence used in generated +recommendations should be returned as `EvidenceReference` objects. + +### Samanth + +Samanth uses these classes as FastAPI request and response models. A minimal +route looks like this: + +```python +from fastapi import APIRouter + +from app.schemas import RiskEngineInput, SafetyIntelligenceResponse + +router = APIRouter() + + +@router.post( + "/intelligence/analyze", + response_model=SafetyIntelligenceResponse, +) +async def analyze_risk( + risk: RiskEngineInput, +) -> SafetyIntelligenceResponse: + ... +``` + +### Rajath + +Rajath builds the frontend against the JSON form of +`SafetyIntelligenceResponse`. The frontend should treat enum strings as the +documented controlled vocabulary and should display limitations and human-review +state rather than hiding them. + +## 6. How FastAPI uses Pydantic schemas + +For incoming requests, FastAPI reads JSON and asks Pydantic to build the declared +request model. Invalid types, values, or unknown fields produce an HTTP 422 +response before endpoint logic runs. + +For outgoing responses, `response_model=SafetyIntelligenceResponse` makes +FastAPI validate and serialize the returned value. This catches incomplete or +malformed service results and prevents accidental fields from leaking through. + +FastAPI also reads the models to generate OpenAPI definitions. Swagger UI then +shows field names, types, enum choices, constraints, and nested models without a +separate handwritten API specification. + +## 7. How to create and validate an object + +Create a model directly from Python values: + +```python +from datetime import datetime, timezone + +from app.schemas import ShiftLogEntry + +log = ShiftLogEntry( + log_id="LOG-001", + timestamp=datetime.now(timezone.utc), + shift_id="DAY-01", + author_role="Operator", + raw_text="Ventilation fan did not restart after the maintenance check.", + acknowledged=False, + resolved=False, +) + +print(log.log_id) +print(log.model_dump_json(indent=2)) +``` + +Validate an existing dictionary or JSON string with Pydantic v2: + +```python +from app.schemas import RiskEngineInput + +risk = RiskEngineInput.model_validate(payload_dictionary) +risk_from_json = RiskEngineInput.model_validate_json(payload_json_string) +``` + +Invalid data raises `pydantic.ValidationError`. Application code may catch that +exception outside FastAPI; FastAPI converts request validation errors to HTTP +422 responses automatically. + +## 8. How to run the tests + +From the repository root, create/activate a virtual environment and install the +development requirements once: + +```bash +python3 -m venv .venv +.venv/bin/python -m pip install -r requirements-dev.txt +``` + +Run the configured complete backend test suite: + +```bash +.venv/bin/python -m pytest +``` + +`pytest.ini` adds `backend` to Python's import path, so tests and production code +both use imports such as `from app.schemas import RiskEngineInput`. The schema +tests also validate every file under `docs/examples`. + +## 9. How to extend the schemas safely + +Prefer a new optional field with a sensible default when old producers will not +send it. Renaming or deleting a field is a breaking change and must be +coordinated with the simulator, risk engine, backend, RAG pipeline, and frontend. + +Add enum values carefully. Consumers may render an exhaustive list or map every +value to behavior. Update tests, JSON examples, and this document in the same +change. Avoid `dict[str, Any]` unless the data is genuinely variable; a named +nested model is easier to document and validate. + +Keep public APIs versionable. If a necessary change cannot remain backward +compatible, introduce a versioned endpoint or model rather than silently +changing the meaning of an existing field. + +The pre-existing `maintainance.py` filename is misspelled. Do not rename it until +all imports and integrations can be updated together. Similarly, translate the +digital twin's legacy zone and lowercase enum values at an explicit ingestion +boundary rather than teaching every downstream consumer multiple spellings. + +## 10. Known limitations + +- Refinery zones are prototype abstractions, not a surveyed plant-area model. +- Sensor ranges do not yet model a specific real refinery or instrument setup. +- Recommendations are advisory and require qualified human review and approval. +- Synthetic incidents must always be clearly labeled with `is_synthetic=true`. +- Schema validation cannot guarantee that an AI recommendation is factually correct or safe. +- Retrieved RAG evidence must still be checked, cited, and verified against controlled source documents. +- The current simulator vocabulary needs an adapter before direct validation against the canonical shared enums. diff --git a/docs/examples/extracted_safety_event.json b/docs/examples/extracted_safety_event.json new file mode 100644 index 0000000..30d7e36 --- /dev/null +++ b/docs/examples/extracted_safety_event.json @@ -0,0 +1,17 @@ +{ + "source_log_id": "LOG-NIGHT-118", + "zone_id": "ZONE_B", + "equipment_ids": ["P-101", "VENT-ZB-01"], + "hazards": [ + "RISING_LEL", + "VENTILATION_FAILURE", + "HOT_WORK_ACTIVE", + "WORKERS_PRESENT", + "UNRESOLVED_SHIFT_OBSERVATION" + ], + "severity": "HIGH", + "confidence": 0.89, + "summary": "An unresolved hydrocarbon observation coincides with failed ventilation and active hot work in Zone B.", + "requires_follow_up": true, + "extraction_method": "RULE_LLM_HYBRID" +} diff --git a/docs/examples/historical_incident.json b/docs/examples/historical_incident.json new file mode 100644 index 0000000..4c29604 --- /dev/null +++ b/docs/examples/historical_incident.json @@ -0,0 +1,34 @@ +{ + "incident_id": "INC-SYN-014", + "title": "Pump seal vapour ignited during nearby hot work", + "occurred_at": "2024-03-18T14:20:00Z", + "industry": "PETROLEUM_REFINERY", + "facility_type": "REFINERY_PROCESS_UNIT", + "zone_id": "ZONE_B", + "equipment_ids": ["P-044", "VENT-PS-02"], + "risk_type": "FIRE_EXPLOSION", + "severity": "CRITICAL", + "permit_types": ["HOT_WORK"], + "contributing_hazards": [ + "RISING_LEL", + "VENTILATION_FAILURE", + "HOT_WORK_ACTIVE", + "WORKERS_PRESENT" + ], + "summary": "Hydrocarbon vapour from a degrading pump seal accumulated after ventilation failed and ignited during permitted hot work.", + "root_causes": [ + "The pump seal leak was not isolated before work began", + "Ventilation failure was not linked to permit suspension" + ], + "consequences": [ + "Flash fire caused worker injuries and pump-area damage" + ], + "preventive_actions": [ + "Continuously correlate gas trends, ventilation state, worker presence, and active hot-work permits", + "Require a supervisor to suspend work when compound risk exceeds the approved threshold" + ], + "source_title": "Synthetic refinery learning case 14", + "source_url": null, + "source_page": 7, + "is_synthetic": true +} diff --git a/docs/examples/risk_engine_input.json b/docs/examples/risk_engine_input.json new file mode 100644 index 0000000..d4fb019 --- /dev/null +++ b/docs/examples/risk_engine_input.json @@ -0,0 +1,36 @@ +{ + "alert_id": "ALT-2026-001", + "timestamp": "2026-07-16T09:30:00Z", + "zone_id": "ZONE_B", + "equipment_ids": ["P-101"], + "risk_type": "FIRE_EXPLOSION", + "risk_score": 0.91, + "severity": "CRITICAL", + "predicted_incident": "Hydrocarbon flash fire near pump P-101", + "contributing_factors": [ + "RISING_LEL", + "HOT_WORK_ACTIVE", + "VENTILATION_FAILURE", + "WORKERS_PRESENT" + ], + "sensor_evidence": { + "hc_gas_lel": { + "sensor_id": "LEL-ZB-04", + "value": 8.4, + "unit": "%LEL", + "trend": "RISING", + "normal_single_sensor_alarm_threshold": 10.0 + }, + "ventilation": { + "equipment_id": "VENT-ZB-01", + "operational": false + }, + "detection_basis": "Compound conditions detected before the normal single-sensor alarm threshold" + }, + "active_permit_ids": ["PTW-HW-204"], + "maintenance_event_ids": ["MNT-VENT-031"], + "shift_log_ids": ["LOG-NIGHT-118"], + "cctv_event_ids": ["CCTV-ZB-778"], + "estimated_lead_time_minutes": 12.0, + "model_confidence": 0.88 +} diff --git a/docs/examples/safety_intelligence_response.json b/docs/examples/safety_intelligence_response.json new file mode 100644 index 0000000..7f9bab6 --- /dev/null +++ b/docs/examples/safety_intelligence_response.json @@ -0,0 +1,106 @@ +{ + "intelligence_id": "INT-2026-001", + "alert_id": "ALT-2026-001", + "generated_at": "2026-07-16T09:30:08Z", + "executive_summary": "Zone B has a critical compound flash-fire risk before the normal hydrocarbon alarm threshold is reached.", + "risk_explanation": "Hydrocarbon concentration is rising around pump P-101 while ventilation is unavailable, a hot-work permit is active, and workers are present. Together these conditions create a credible ignition pathway even though the gas detector has not reached its normal standalone alarm threshold.", + "recommended_actions": [ + { + "action_id": "ACT-001", + "title": "Review immediate suspension of hot work", + "description": "The area supervisor should verify conditions and, if confirmed, suspend the hot-work permit under the site's approved permit procedure.", + "priority": 10, + "status": "PROPOSED", + "requires_human_approval": true, + "target_role": "Area Supervisor", + "supporting_evidence_ids": ["EVD-SENSOR-001", "EVD-PERMIT-001", "EVD-SOP-001"] + }, + { + "action_id": "ACT-002", + "title": "Account for exposed workers", + "description": "The shift supervisor should confirm worker locations and apply the approved site evacuation or exclusion procedure if conditions warrant it.", + "priority": 9, + "status": "PROPOSED", + "requires_human_approval": true, + "target_role": "Shift Supervisor", + "supporting_evidence_ids": ["EVD-CCTV-001", "EVD-SOP-001"] + } + ], + "evidence": [ + { + "evidence_id": "EVD-SENSOR-001", + "evidence_type": "SENSOR", + "title": "Zone B hydrocarbon trend", + "excerpt": "LEL-ZB-04 is rising at 8.4 percent LEL while the normal standalone alarm threshold is 10 percent LEL.", + "source_name": "SCADA historian", + "source_url": null, + "page_number": null, + "section_name": "LEL-ZB-04 recent readings", + "relevance_score": 0.98 + }, + { + "evidence_id": "EVD-PERMIT-001", + "evidence_type": "PERMIT", + "title": "Active Zone B hot-work permit", + "excerpt": "Permit PTW-HW-204 is active for hot work near pump P-101.", + "source_name": "Permit-to-work service", + "source_url": null, + "page_number": null, + "section_name": "Permit status", + "relevance_score": 0.97 + }, + { + "evidence_id": "EVD-CCTV-001", + "evidence_type": "CCTV", + "title": "Workers detected near P-101", + "excerpt": "Structured CCTV event CCTV-ZB-778 reports three workers in the affected work area.", + "source_name": "CCTV event pipeline", + "source_url": null, + "page_number": null, + "section_name": "Worker detection event", + "relevance_score": 0.94 + }, + { + "evidence_id": "EVD-SOP-001", + "evidence_type": "SOP", + "title": "Hot-work atmospheric safety procedure", + "excerpt": "The retrieved procedure requires supervisory review when ventilation or atmospheric conditions become unsafe during hot work.", + "source_name": "Prototype Refinery SOP Library", + "source_url": "https://example.invalid/sop/hot-work", + "page_number": 12, + "section_name": "Changing atmospheric conditions", + "relevance_score": 0.92 + } + ], + "similar_incidents": [ + { + "incident_id": "INC-SYN-014", + "title": "Pump seal vapour ignited during nearby hot work", + "similarity_score": 0.89, + "shared_hazards": [ + "RISING_LEL", + "VENTILATION_FAILURE", + "HOT_WORK_ACTIVE", + "WORKERS_PRESENT" + ], + "summary": "Hydrocarbon vapour accumulated after ventilation failed and ignited during permitted hot work.", + "root_causes": [ + "The pump seal leak was not isolated before work began", + "Ventilation failure was not linked to permit suspension" + ], + "preventive_actions": [ + "Correlate gas trends, ventilation state, workers, and active permits" + ], + "source_title": "Synthetic refinery learning case 14", + "source_url": null, + "source_page": 7 + } + ], + "intelligence_confidence": 0.9, + "insufficient_evidence": false, + "limitations": [ + "This prototype has not verified readings against a production refinery instrument system", + "The retrieved historical incident is synthetic and must not be presented as a real event" + ], + "requires_human_review": true +} diff --git a/docs/examples/shift_log_entry.json b/docs/examples/shift_log_entry.json new file mode 100644 index 0000000..3cefda0 --- /dev/null +++ b/docs/examples/shift_log_entry.json @@ -0,0 +1,11 @@ +{ + "log_id": "LOG-NIGHT-118", + "timestamp": "2026-07-16T09:24:00Z", + "shift_id": "SHIFT-NIGHT-2026-07-16", + "author_role": "Console Operator", + "zone_id": "ZONE_B", + "equipment_ids": ["P-101", "VENT-ZB-01"], + "raw_text": "Faint hydrocarbon odour reported near P-101. Ventilation fan did not restart; hot-work crew remains in the area.", + "acknowledged": true, + "resolved": false +} diff --git a/pytest.ini b/pytest.ini new file mode 100644 index 0000000..e071ebe --- /dev/null +++ b/pytest.ini @@ -0,0 +1,3 @@ +[pytest] +pythonpath = backend +testpaths = backend/tests diff --git a/requirements-dev.txt b/requirements-dev.txt new file mode 100644 index 0000000..1b79ad3 --- /dev/null +++ b/requirements-dev.txt @@ -0,0 +1,3 @@ +-r requirements.txt + +pytest==9.0.2 diff --git a/requirements.txt b/requirements.txt index 9d0591d..91bc3b1 100644 --- a/requirements.txt +++ b/requirements.txt @@ -15,3 +15,9 @@ starlette==1.3.1 typing-inspection==0.4.2 typing_extensions==4.16.0 uvicorn==0.51.0 +# Optional for extracting text from authorized, text-based PDFs. Scanned PDFs +# still require a separate manual/OCR workflow and are not processed here. +pypdf>=4.0,<6 +# Optional local embedding adapter. Model weights are loaded separately by the +# user and are never downloaded by the test suite. +sentence-transformers>=3.0,<6